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Abstract 

In  the  critical  arena  of  source  selections,  the  decision¬ 
maker  is  often  overwhelmed  by  the  complex  hierarchy  of 
intertwining  factors  and  multitude  of  conflicting  tasks 
required  to  successfully  purchase  an  effective  weapons 
system.  At  this  time,  there  is  no  available  tool  with  which 
to  assimilate  contributing  criteria  into  an  organized 
framework  to  aid  in  the  decision  making  process.  A  decision 
support  system  (DSS)  acts  as  the  framework  upon  which  the 
complex  elements  may  be  organized.  The  purpose  of  this 
research  is  to  test  the  use  of  a  computer-aided  decision 
support  system  in  the  source  selection  environment. 

Through  a  controlled  experiment,  the  use  of  a  DSS  was  tested 
for  the  following  variables. 

1.  Effectiveness.  This  was  defined  as  the  number  of 
"correct"  decisions  made. 

2.  Consistency.  This  was  defined  as  how  many  of  the 
same  decisions  were  made. 

3.  Speed.  Did  the  use  of  the  DSS  speed  the  process? 

4.  Difficulty.  How  easy  was  it  to  make  the  decision? 

5.  Confidence.  How  confident  was  the  decision-maker 
that  his  decision  was  correct? 
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6.  Understanding.  How  well  did  the  subject  understand 
the  process  by  which  the  decision  was  made? 

The  results  of  the  experiment  indicated  little  effect  of 
the  DSS  on  effectiveness  and  consistency.  A  negative 
correlation  was  discovered  between  the  use  of  the  DSS  and 
the  time  required  to  reach  a  decision.  A  positive 
correlation  was  discovered  between  DSS  and  the  variables 
ease,  confidence  and  understanding. 

These  results  suggest  further  research  into  the 
applications  of  decision  support  systems.  In  the  Air  Force 
environment  where  any  decision  must  pass  multiple  approval 
"tests,"  the  perceived  increase  in  confidence  and 
understanding  would  certainly  be  an  advantage  for  the 
decision  maker.  The  increased  simplicity  of  the  process 
speaks  for  itself. 
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THE  USE  OF  COMPUTER-AIDED  DECISION 
SUPPORT  SYSTEMS  FOR  COMPLEX 
SOURCE  SELECTION  DECISIONS 


I .  Introduction 


One  of  the  most  complex  decisions  made  in  the 
government  is  that  of  the  source  selection  decision.  The 
acquisition  of  Department  of  Defense  (DOD)  weapons  systems 
is  made  up  of  many  complex  tasks  (Thybony,  1988:8).  It  is 
the  large  scope  of  necessary  activities  and  the  intricate 
interrelationships  of  the  factors  involved  to  ultimately 
achieve  a  source  selection  decision  that  warrant  the 
investigation  into  an  effective  computer-aided  decision 
support  system. 

The  multiple  tasks  involved  are  divided  among  a  variety 
of  specialists  in  their  various  functions,  but  are  under  the 
overall  control  of  the  program  manager  and  the  contracting 
officer.  While  the  program  manager  has  the  responsibility 
to  achieve  overall  program  success  (FAR:  1.6,  2.1,  15.604), 
the  contracting  officer  has  the  authority  to  legally  bind 
the  government  in  a  contractual  document. 

Although  these  two  individuals  carry  the  brunt  of  the 
load,  it  is  the  Source  Selection  Authority  (SSA)  who  must 
make  the  final  decision  as  to  which  contractor  will  receive 
the  final  award  (Cibinic  et.  al.,  1988:1-1).  This  decision 


1 


must  be  a  compilation  of  the  efforts  of  the  functional 
specialists  towards  the  accomplishment  of  a  lofty  goal:  to 
acquire  a  timely  system  that  is  responsive  to  the  needs  of 
the  user,  and  to  acquire  it  at  a  reasonable  price. 

The  contracting  officer  and  the  program  manager  must 
make  award  decisions  based  on  a  complex  hierarchy  of 
intertwined  factors  on  a  daily  basis.  Faced  with  these 
intricate  relationships  and  the  multitude  of  conflicting 
tasks  ,  the  contracting  officer  and  the  program  manager 
could  easily  be  overwhelmed. 

The  ultimate  decision  involves  not  only  multiple 
criteria,  but  it  also  involves  many  alternative  approaches 
to  the  problem  at  hand  to  choose  from.  In  such  a  situation, 
the  decision-making  process  can  easily  become  confused. 

With  dozens  of  competing  criteria  and  sub-criteria, 
decisions  may  not  take  into  account  all  relevant  factors. 
More  likely,  the  SSA  will  make  a  decision  without  fully 
understanding  the  relative  importance  of  the  interaction  of 
these  criteria. 

If  the  decisions  do  not  reflect  the  interaction  of  the 
competing  factors,  a  real  threat  is  posed  to  the  entire 
decision-making  process.  Systems  that  do  not  perform  up  to 
expectations,  or  are  unreasonably  costly  may  be  purchased. 
The  results  of  such  purchases  could  prematurely  make 
obsolete  the  system  being  procured,  with  the  worst  case 


2 


being  that  the  defense  of  the  nation  is  adversely  affected. 
The  search  for  an  effective  aid  to  the  decision-making 
process  is  warranted  for  all  of  these  reasons. 

The  complex  environment  of  source  selections  requires  a 
new  way  to  cope  with  the  multitude  of  factors  that  affect 
the  decisions  that  are  made.  Because  the  system  proposed 
must  be  understandable  to  those  who  will  employ  it,  yet  not 
be  so  simplistic  as  to  be  useless,  the  new  decision-making 
process  must  be  logical,  simple  and  should  just  plain  appeal 
to  "good  sense." 

This  research  proposes  a  decision  support  system  based 
on  the  Analytical  Hierarchy  Process  to  aid  the  program 
manager,  the  contracting  officer  and  the  SSA  in  identifying 
and  evaluating  all  the  relevant  criteria  and  alternatives  in 
making  the  final  source  selection  decision.  The  issue  to  be 
addressed  by  this  research  is  not  whether  the  decisions 
currently  being  made  are  right  or  wrong,  but  whether  they 
can  be  improved. 

In  order  for  a  computer-aided  support  system  to  be 
effective  in  source  selections,  the  individual  activities 
leading  to  the  decision  must  be  reviewed.  Early  in  the 
source  selection  process  a  Source  Selection  Plan  is 
formulated  in  order  to  delineate  the  areas  that  will  be 
important  to  award.  These  areas  are  split  into  factors  and 
subsequently,  the  factors  into  items.  Each  of  the  items. 
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factors,  and  areas  relate  to  each  other  in  a  different 
fashion  and  have  different  relative  weights  (Cibinic  et. 
al.,  1988:V-12).  The  ability  to  assimilate  this  information 
into  an  understandable  decision  is  paramount  to  the  process 
and  can  only  be  worsened  by  ever  present  time  constraints. 
Thus  an  organized  framework  is  needed  to  keep  up  with  the 
interaction  of  the  criteria  contributing  to  the  decision. 

This  research  addresses  the  problem  that  at  this  time 
there  is  no  tool  to  aid  the  program  managers  and  contracting 
officers  in  their  assimilation  of  the  criteria  as  they  make 
these  complex  decisions  on  a  daily  basis.  To  further 
complicate  the  matter,  the  final  decision  is  made  by  the  SSA 
who  is  not  directly  involved  in  the  evaluation  of  the 
competing  proposals;  thus,  any  mistakes  made  by  the 
contracting  officer  or  the  program  manager  are  magnified  in 
the  final  decision. 

A  less  than  desirable  decision  in  the  Department  of 
Defense  can  have  far-reaching  repercussions;  a  system  may 
not  be  fielded  in  a  timely  manner;  it  may  not  address  the 
threat  that  it  was  designed  to  counter;  it  may  be  too  costly 
to  be  an  effective  system.  Clearly,  these  consequences 
constitute  a  danger  that  the  best  possible  decisions  are  not 
being  made  by  the  government  in  the  source  selection 
process.  This  danger  may  be  lessened  by  the  development  of 
a  decision  support  system  (DSS)  to  aid  in  the  process. 
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For  all  of  the  aforementioned  reasons,  the  search  for 
an  effective  aid  to  the  decision-making  process  is  both 
timely  and  necessary.  It  is  the  search  for  an  effective  aid 
to  the  decision-making  process  on  which  this  research  is 
focused. 

Specific  Problem 

Given  the  limited  capability  of  the  human  mind  to 
process  large  amounts  of  information  in  complex  situations, 
there  is  a  high  probability  that  the  best  decisions  are  not 
being  made  in  the  DOD  acquisition  process.  Many  authors 
have  questioned  the  ability  of  humans  as  information 
processors.  Most  notably,  Miller  proposes  a  theory  based  on 
the  "magic  number  seven,  plus  or  minus  two."  The  theory  is 
described  by  Davis  and  Olson:  the  human  mind  is  only  capable 
of  tracking  seven  (plus  or  minus  two)  pieces  of  information 
at  one  time  (Davis  and  Olson,  1985:245). 

The  purpose  of  this  research  is  to  search  for  an 
effective  aid  in  the  decision-making  process.  The  obvious 
problem  with  the  outcome  is  encouraging  the  use  of  the 
system.  There  is  a  fear  among  existing  workers  that  the 
computer  is  making  the  decisions.  The  adoption  of  a  DSS  may 
be  viewed  as  allowing  a  computer  to  make  the  most  important 
decision  in  the  acquisition  process.  However,  a  DSS  is  only 
a  tool;  it  does  not  make  a  decision,  it  merely  aids  in  the 
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organization  and  the  assimilation  of  the  multiple  factors 
that  go  into  the  decision-making  process. 

The  acquisition  field  is  ripe  for  the  introduction  of 
some  decision-making  aid.  With  billions  of  dollars  at 
stake,  the  decision-maker  should  have  any  advantage  that  he 
can  get  in  order  to  make  his  final  choice.  Given  the 
limited  capability  of  the  human  mind  to  understand  and 
assimilate  the  large  amounts  of  information  involved  in  the 
acquisition  process,  there  is  a  high  probability  that  the 
quality  of  the  decisions  being  made  can  be  improved  upon. 
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II .  Literature  Review 


This  chapter  is  divided  into  three  sections,  1) 
Decision-Making,  2)  The  Analytical  Hierarchy  Process,  and  3) 
The  Expert  Choice  Software.  The  section  on  decision-making 
will  address  classical  decision  theory  and  the  theory  of 
Decision  Support  Systems  (DSS) .  The  second  section  will  be 
devoted  to  Thomas  Saaty  and  his  Analytical  Hierarchy 
Process.  Finally,  the  Expert  Choice  DSS  will  be  briefly 
described. 

Decision-Making 

Decision-making  is  loosely  defined  as  choosing  between 
alternatives;  thus  the  primary  role  of  managers  today  is  to 
make  decisions.  Through  the  proper  decisions,  organizations 
are  either  successful  or  unsuccessful  in  achieving  their 
goals;  therefor,  decision-making  is  of  paramount  importance 
to  any  organization.  Not  surprisingly  a  large  body  of 
literature  has  been  devoted  to  decisions  and  decision-making 
theory;  the  most  well-known  and  accepted  of  the  theorists 
is  Herbert  A.  Simon. 

Simon  warns  against  the  simplified  definition  of 
decision-making  presented  above  when  he  describes  a 
"lengthy,  complex  process  of  alerting,  exploring  and 
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analyzing  that  precede  that  final  moment”  of  choice  (Simon, 
1977:40).  In  his  book,  The  New  Science  of  Management 
Decision .  Simon  examines  the  decision-maker  in  all  the 
phases  of  the  decision-making  process  when  he  describes  his 
classic  model:  1)  Intelligence,  2)  Design,  3)  Choice,  and 
4)  Review  (Simon,  1977:40).  The  model  suggests  that  there 
is  a  natural  progression  from  one  phase  to  the  next, 
allowing  for  the  return  to  any  previous  phase  if  the 
situation  warrants  (Simon,  1977:43). 

Intelligence.  During  this  phase  the  decision-maker 
identifies  a  problem,  defines  and  refines  the  problem  and 
decides  if  action  is  necessary.  Intelligence  activities  are 
how  the  decision-maker  recognizes  dissatisfaction  with  the 
current  state  of  affairs. 

Design.  The  design  phase  consists  of  searching  for, 
creating,  and  analyzing  alternative  courses  of  action.  This 
process  includes  testing  for  the  feasibility  of  any  possible 
solutions . 

Choice.  This  is  the  time  when  a  choice  among 
alternatives  is  made.  Simon  claims  that  most  definitions  of 
decision-making  concentrate  on  only  this  phase  of  the 
process  and  ignore  the  other  phases  of  decision-making. 

Review.  The  decision  is  implemented  and  reviewed  at 
this  stage  (Simon,  1977:40-42). 
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Simon's  model  asserts  that  although  all  decisions  are 

made  through  a  process  similar  to  the  one  described  above, 

the  process  is  affected  by  the  type  of  decision  to  be  made. 

Simon  separates  decisions  into  two  categories:  programed 

and  nonprogrammed  (Simon,  1977:45).  Simon  states: 

Decisions  are  programed  to  the  extent  that  they 
are  repetitive  and  routine,  to  the  extent  that  a 
definite  procedure  has  been  worked  out  for 
handling  them  so  that  they  don't  have  to  be 
treated  d*  novo  each  time  they  occur  (Simon, 

1977:46) . 

Simon  goes  on  to  differentiate  these  from  nonprogrammed 
decisions. 

Decisions  are  nonprogrammed  to  the  extent  that 
they  are  novel,  unstructured  and  usually 
consequential .  There  is  no  cut-and-dried  method 
for  handling  the  problem  because  it  hasn't  arisen 
before,  or  because  its  precise  nature  and 
structure  are  elusive  or  complex,  or  because  it  is 
so  important  that  it  deserves  a  custom-tailored 
treatment  (Simon, 1977 : 46) . 

This  separation  into  programed  and  nonprogrammed 
decisions  is  supported  by  Lohaus  who  describes  "programmed" 
and  "custom"  decisions.  Lohaus'  programmed  decisions  are 
those  that  are  recurring  and  based  on  established  policies 
or  routines  (Lohaus,  1985:3.17.7).  Custom  decisions  are 
those  that  require  the  personal  attention  of  the 
professional  manager  because  the  decision  must  be  customized 
to  fit  the  specific  situation  (Lohaus , 1985 : 3 . 17 . 7) . 

Further  support  is  lended  by  Gorry  and  Scott  Morton  who 
describe  "structured"  and  "unstructured"  decisions. 
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Integral  to  their  position  is  the  thought  that  the 

unstructured  decision  requires  the  human  decision-maker  to 

make  subjective  judgments,  whereas  structured  decisions  do 

not  (Gorry  and  Scott  Morton,  1971:60). 

Key  to  all  of  these  positions  is  that  the 

"nonprogrammed"  decision  is  not  repetitive  or  recurring  like 

the  ,,programmed,,  decision.  Simon  suggests  that  if  a  problem 

occurs  often  enough,  a  routine  will  be  developed  to  address 

it,  thus  making  the  problem  "programed”  (Simon,  1977:46). 

Similarly,  Sprague  and  Carlson  suggest  that  decisions 

are  "unstructured"  due  to  uncertainty  (Sprague  and  Carlson, 

1982:94).  In  either  event,  while  the  very  nature  of  the 

"nonprogrammed"  decision  requires  special  attention,  the 

recurring  aspect  of  the  "structured"  problem  suggests  that 

routines  can  be  developed  to  make  the  decision. 

As  the  review  above  indicates,  decisions  fall  into  one 

of  two  categories:  programmed  or  nonprogrammed.  Simon  uses 

this  distinction  to  support  that  different  processes  are 

used  to  make  different  types  of  decisions.  Simon  writes: 

The  main  reason  for  distinguishing  between 
programed  and  nonprogrammed  decisions  is  that 
different  techniques  are  used  for  handling  these 
two  aspects  of  our  decision  making.  The 
distinction,  then,  will  be  a  convenient  one  for 
classifying  these  techniques.  I  shall  use  it  for 
that  purpose,  hoping  that  the  reader  will  remind 
himself  from  time  to  time  that  the  world  is  mostly 
gray  with  only  a  few  patches  of  pure  black  or 
white  (Simon,  1977:47). 
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Cook  summarizes  the  review  quite  well  when  he  writes: 


.  .  .  decision  types  can  be  classified  into  two 
general  categories: 

1.  Programmed:  Structured,  routine, 
repetitive,  governed  by  procedure,  operational, 
non-complex,  etc. 

2.  Nonprogrammed:  Unstructured, 
underspecified,  unique,  novel,  creative, 
strategic,  complex,  etc. 

While  a  given  decision  may  fall  into  the 
"gray  area"  between  the  two  general  types,  the 
preponderance  of  writing  in  this  field  and  the 
variety  of  nomenclature  used  by  different  authors 
to  refer  to  essentially  the  same  decision  types 
permits  use  of  the  two  categories  as  a  reasonable 
basis  for  analysis  (Cook,  1987:21). 

The  types  of  decisions  have  been  categorized;  the 

processes  through  which  decisions  are  made  must  also  be 

examined  categorically.  While  not  always  referred  to  by  the 

same  nomenclature,  two  major  processes  reveal  themselves  in 

the  literature;  optimizing,  and  satisficing  are  the 

decision-making  processes  under  which  virtually  all 

decisions  can  be  categorized. 

Optimizing.  To  optimize  is  to  make  the  best  possible 

decision  under  the  circumstances  at  hand.  Simon's  process 

for  finding  the  "optimal"  solution  is  that  of  a  rational  man 

making  his  decision  based  on  perfect  knowledge  of  all 

alternatives;  alternatives  that  are  then  subjected  to  an 

economic  cost-benefit  analysis  (Simon,  1976 : xxviii) . 
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Lohaus  describes  optimizing  in  a  similar  fashion;  the 
decision-maker  considers  all  possible  solutions  by  weighing 
the  alternatives  against  each  other  until  the  best  or 
optimal  solution  is  found  (Lohaus,  1985:3.17.8). 

Kepner  and  Tregoe  assert  a  theory  similar  to  that  of 
Simon,  defining  an  optimizing  decision  analysis  as  a  system 
through  which  the  decision-maker  comes  to  the  optimal 
decision.  The  decision-maker  considers  the  ’'musts”  and  the 
"wants”  in  the  context  of  the  possible  alternatives.  By 
performing  analysis  on  these  competing  alternatives,  the 
decision  is  made  (Kepner  and  Tregoe,  1981:86-88). 

Optimizing  is  indeed  a  lofty  goal.  In  fact,  Simon  is 
skeptical  that  it  can  be  achieved  by  the  unaided  human  mind. 
Many  theorists  have  discussed  the  limitations  inherent  to 
the  human  mind.  These  limitations  will  be  discussed  later 
in  the  context  of  optimizing  and  satisficing. 

Satisficing.  Satisficing  is  the  act  of  finding  the 
solution  that  is  "good  enough."  Lohaus  describes 
satisficing  as  identifying  a  solution,  testing  the  solution, 
and  securing  the  acceptance  of  the  players  that  are  involved 
(Lohaus, 1985: 3 . 17 . 8) .  Because  every  possible  solution 
cannot  be  enumerated  and  subjected  to  cost-benefit  analysis, 
the  decision-maker  settles  for  an  "acceptable"  solution. 
Thus,  satisficing  is  how  the  human  mind  copes  with  its  own 
limitations . 
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Simon  further  describes  satisficing  by  listing  habit, 
clerical  routine  and  organizational  structure  as  being 
prevalent  decision-making  aids  (Simon,  1977:48).  These 
methods  all  exhibit  the  characteristics  of  satisficing  as 
they  are  based  on  established  routines  coming  to  a 
satisfactory  solution.  An  obvious  danger  to  these  methods, 
is  that  the  established  routines  may  lag  behind  the  dynamic 
environment  of  the  decision-maker  and  become  obsolete.  If 
this  is  the  case  then  not  only  is  the  optimal  solution  not 
achieved,  but  a  satisfactory  solution  is  not  achieved. 

Many  authors  have  described  the  human  decision  process 
as  some  form  of  satisficing.  By  describing  an 
"administrative”  man  who  looks  for  a  solution  that  is 
satisfactory,  Simon  introduces  his  theory  of  "bounded 
rationality"  in  order  to  explain  how  the  decision-maker 
coped  with  overwhelming  complexity.  (Simon,  1976:xxix) . 

Likewise,  Lindblom  describes  his  "science  of  muddling 
through"  as  a  method  that  managers  turn  to  because  it  is 
impossible  to  optimize.  Citing  time  pressures,  budget 
constraints  and  the  ability  to  process  information  as  the 
factors  that  force  the  decision-maker  to  "muddle  through," 
Lindblom  asserts  that  the  decision  maker  turns  to 
satisficing  (Lindblom,  1959:80). 

In  the  same  vein  as  Simon  and  Lindblom,  Soelberg 
presents  his  "implicit  favorite"  model  to  describe  the 
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decision-making  process.  This  model  suggests  that  decision¬ 
makers  are  forced  to  simplify  the  process  because  of 
complexity.  Suggesting  that  the  model  suggests  that  the 
decision-maker  does  not  enter  into  the  "choice"  phase  of 
Simon's  model  until  an  "implicit  favorite"  is  identified, 
Soelberg  maintains  that  the  rest  of  the  decision-making 
process  is  devoted  to  justifying  the  implicit  favorite 
(Soelberg,  1967:19-29). 

Clearly,  decision-making  processes  can  be  categorized 
as  either  "optimizing"  or  "satisficing."  It  can  also  be 
seen  that  given  the  choice,  the  optimal  decision  is 
preferable  to  the  satisfactory  decision.  The  review  above 
supports  that  few  managers  optimize.  Cook  found  that  only 
45.5%  of  managers  optimize  (Cook,  1987:155).  Janis  and  Mann 
summarize  the  obstacles  to  optimizing  into  two  categories: 

1.  Limitations  of  the  human  mind. 

2.  Bureaucratic  obstacles  (Janis  and  Mann,  1977:41). 

This  research  attempts  to  address  the  first  of  the  two 

obstacles  through  the  use  of  decision-support  systems. 

Decision  support  systems  (DSS)  are  computer  programs 
designed  to  aid  in  the  decision-making  process  and  have  been 
described  as  "another  powerful  weapon  in  the  information 
technology  arsenal,  aimed  at  improving  the  effectiveness  of 
managers  in  organizations"  (Fick  and  Sprague,  1980:7).  The 
emphasis  should  be  that  the  DSS  is  a  tool  rather  than  the 
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decision-maker  (Alter,  1980:1).  By  allowing  the  computer  to 
do  what  the  human  mind  cannot,  the  process  is  improved. 

As  the  role  of  the  computer  is  integral  to  the  theory 
of  DSS's,  certain  assumptions  are  made  about  that  role: 

1.  The  computer  must  only  support  the  decision-making 
process.  It  cannot  make  the  decision  itself. 

2.  DSS's  lend  themselves  best  to  "semi-structured" 
problems  where  the  computer  can  augment  the  decision-maker's 
own  judgments  about  the  problem. 

3.  The  decision-maker  and  the  DSS  must  be  in  an 
interactive  mode  to  gain  the  full  advantage  of  the  DSS 
(Keen,  1976) . 

All  of  these  assumptions  show  that  the  decision  itself  is 
made  by  the  manager.  The  DSS  only  provides  support  for  the 
manager. 

Alter  describes  seven  classes  of  DSS's: 

1.  "File  Drawer  Systems."  These  are  basically  online 
filing  systems  or  databases. 

2.  "Data  Analysis  Systems."  These  allow  simple 
analysis  to  be  performed  on  data.  Automated  budget  systems 
fall  into  this  category. 

3.  "Analysis  Information  Systems."  These  add  a 
limited  modelling  capability  to  data  analysis  systems. 

4.  "Accounting  Models."  These  actually  calculate  the 
consequences  of  financial  decisions  for  comparison  purposes. 
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5. 


"Representational  Models." 


This  is  the  class  of 


simulation  models.  These  models  are  used  to  predict 
outcomes  in  uncertain  environments. 

6.  "Optimization  Models."  These  models  generate  the 
"best  solution"  based  on  a  series  of  constraints. 

7.  "Suggestion  Models."  These  models  provide 
decisions  to  repetitive  (structured/programmed)  situations 
(Alter,  1980:90-91). 

While  all  of  the  above  DSS ' s  are  available  to  the 
manager,  it  is  the  optimization  model  that  this  research  is 
concerned  with.  These  models  allow  the  decision-maker  to 
input  his  own  judgments  and  the  computer  can  optimize  the 
solution. 

The  Analytical  Hierarchy  Process 

As  seen  in  the  review  above,  a  number  of  authors  are 
skeptical  about  the  ability  of  the  human  mind  to  "optimize." 
Saaty  has  proposed  a  method  of  optimization  that  will  enable 
the  decision-maker  to  avoid  many  of  the  short-comings  that 
hinder  the  optimization  process. 

According  to  Saaty,  a  complex  decision  is  one  where 
there  are  a  number  of  alternatives  which  are  supported  by 
multiple  decision  criteria  and  other  factors.  These  other 
factors  may  be  tangible  or  intangible  in  nature,  but 


they  interact  to  support  the  overall  decision  (Saaty, 
1982:27)  . 

To  make  the  optimal  decision,  all  possible  alternatives 
must  be  identified  and  considered,  and  all  relevant  criteria 
that  support  the  alternatives  must  be  examined.  In  order  to 
make  sense  of  all  this  information,  some  support  framework 
must  be  employed.  The  Analytical  Hierarchy  Process  provides 
just  such  a  framework: 

Basically  AHP  is  a  method  of  breaking  down  a 
complex,  unstructured  situation  into  its  component 
parts;  assigning  these  parts,  or  variables,  into  a 
hierarchic  order;  assigning  numerical  values  to 
subjective  judgments  on  the  relative  importance  of 
each  variable;  and  synthesizing  the  judgments  to 
determine  which  variables  have  the  highest 
priority  and  should  be  acted  upon  to  influence  the 
outcome  of  the  situation.  (Saaty,  1882:5) 

Saaty  claims  that  the  AHP  represents  the  natural 

decision-making  process  of  the  human  mind  and  allows  for  the 

expansion  of  the  boundaries  of  the  mind  when  dealing  with 

large  amounts  of  information.  The  process  mandates  a 

"rational”  framework  when  dealing  with  multiple  conflicting 

alternatives  and  criteria.  The  AHP  allows  for  the 

structuring  of  the  decision  into  hierarchies,  thus  lending  a 

structure  to  an  otherwise  unstructured  problem  and  giving 

the  decision-maker  a  new  perspective  on  the  problem.  By 

conceiving  of  the  problem  as  a  "whole"  made  up  of  smaller 

parts,  the  interaction  among  criteria  and  alternatives  can 
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be  considered  simultaneously  and  the  optimal  decision  can  be 
made  (Saaty,  1982:24). 

By  claiming  that  this  process  is  innate  to  the  human 
mind,  Saaty  asserts  that  when  faced  with  a  complex  decision, 
the  individual  constructs  a  hierarchy  of  criteria  and 
alternatives  building  up  to  the  decision.  By  assigning 
values  of  importance  to  the  factors  and  making  mental 
judgments  about  the  criteria,  the  decision-making  matrix  is 
established.  The  decision-maker  then  performs  a  mental 
synthesis  on  the  hierarchy  and  selects  the  alternative  that 
is  the  "best"  per  his  own  judgments.  The  AHP  allows  the 
decision-maker  to  look  at  only  parts  of  the  decision  at  one 
time  while  keeping  perspective  as  to  the  whole. 

In  order  to  illustrate  the  AHP,  Saaty  uses  the  example 
of  the  possible  urbanization  of  a  river  region  in 
Pennsylvania.  The  most  important  issue  to  the  decision¬ 
maker  was  the  environmental  quality,  thus  the  protection  of 
the  environmental  quality  becomes  the  top  level  of  the 
hierarchy  (see  figure  1) .  Saaty  goes  on  to  describe  the 
example: 

The  highest  level  was  the  overall  objective  of 
protecting  environmental  quality.  The  lowest 
included  the  final  actions,  or  alternative  plans, 
that  would  contribute  positively  or  negatively  to 
the  main  objective  through  their  impact  on  the 
intermediate  criteria.  The  alternatives  were  (A) 
to  leave  the  area  nonurbanized,  (B)  to  allow 
partial  urbanizations,  and  (C)  to  allow  total 
urbanization.  The  intermediate  levels  of  the 
hierarchy  comprised  the  two  basic  criteria  for 
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evaluating  environmental  quality:  (1)  esthetic 
criteria,  which  were  further  structured  into 
properties  of  vividness,  intactness,  and  no  noise 
or  disturbances;  and  (2)  hydrologic  criteria, 
subdivided  into  no  flooding,  water  quality,  and 
channel  naturalness.  This  hierarchy  graphically 
depicts  the  interdependence  of  elements  in  the 
problem;  it  both  isolates  the  relevant  factors  and 
displays  them  in  the  larger  context  of  their 
relationship  to  each  other  and  to  the  system  as  a 
whole  (Saaty,  1982:15-16). 


The  overall  hierarchy  of  the  decision  is  shown  in 
figure  1. 


ENVIRONMENTAL  QUALITY 


ESTHETICS  HYDROLOGY 


IVIDNESS  INTACTNESS  NOISE  FLOODING  WATER  QUALITY  CHANNE 


NONURBANIZED  PARTLY  URBANIZED  TOTALLY  URBANIZED 


Figure  1  Adapted  from  Figure  2-1  (Saaty,  1982:15) 
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After  constructing  the  hierarchy,  it  is  incumbent  upon 
the  decision-maker  to  assign  relative  importance  to  each  of 
the  factors  within  the  hierarchy.  In  the  AHP,  these 
judgments  are  quantified  in  order  to  compare  them.  As  Saaty 
states,  "Often  words  alone  or  logical  argument  cannot 
express  the  subtleties  of  deeply  felt  differences"  (Saaty, 
1982:16) . 

The  next  step  in  the  process  is  to  make  judgments  on 
the  different  merits  of  the  alternatives.  These  judgments 
are  made  at  the  lowest  level  of  the  hierarchy  thus  allowing 
for  the  most  simple  judgement  to  be  made  by  the  decision¬ 
maker.  The  AHP  simplifies  this  process  by  structuring  the 
problem  into  sections  that  the  decision-maker  can  handle. 
While  the  entire  problem  at  once  may  be  overwhelming,  humans 


have  the  ability  to  perceive  relationships  among 
the  things  they  observe,  to  compare  pairs  of 
similar  things  against  certain  criteria,  and  to 
discriminate  between  both  members  of  a  pair  by 
judging  the  intensity  of  their  preference  for  one 
over  the  other.  Then  they  synthesize  their 
judgments  —  through  imagination  or,  with  the  AHP, 
through  a  new  logical  process  —  and  gain  a  better 
understanding  of  the  whole  system  (Saaty, 

1982:17) . 


The  next  issue  at  hand  is  that  of  consistency.  If  the 
judgments  are  not  consistent,  then  the  decision  is  flawed 
and  the  judgments  should  be  reconsidered,  thus  the  AHP  tests 
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for  consistency  of  judgments.  (Saaty,  1982:16).  As  Saaty 
states: 

The  consistency  is  perfect  if  all  the  judgments 
relate  to  each  other  in  a  perfect  way.  If  you  say 
that  you  prefer  spring  to  summer  three  times  more 
and  that  you  prefer  summer  to  winter  twice  more, 
then  when  you  give  the  judgement  comparing  your 
preference  of  spring  to  winter  it  should  be  6  and 
not  anything  else.  The  greater  your  deviation 
from  6,  the  greater  your  inconsistency.  This 
observation  applies  to  relations  among  all  the 
judgments  given.  We  would  have  perfect 
consistency,  then,  if  all  the  relations  checked 
out  correctly  (Saaty,  1982:16). 


The  validity  of  the  Analytical  Hierarchy  Process  is 
defended  by  Harker  and  Vargas  in  their  paper  "The  Theory  of 
Ratio  Scale  Estimation:  Saaty' s  Analytical  Hierarchy 
Process"  when  they  state,  "AHP  is  based  on  a  firm 
theoretical  foundation  and,  as  examples  in  the  literature 
.  .  .  illustrate,  the  AHP  is  a  viable,  usable  decision¬ 
making  tool"  (Harker  and  Vargas,  1985:50). 

This  research  proposes  to  employ  the  AHP  in  the 
decision-making  process  through  the  use  of  a  DSS.  Expert 
Choice,  developed  by  Decision  Support  Software  will  be  used 
by  the  author  to  construct  and  test  a  model  that  will  aid  in 
the  source  selection  process. 


The  Expert  Choice  Software 

Expert  Choice  is  a  decision  support  program  that  is 
based  on  Saaty 's  AHP.  The  system  allows  the  user  to 


graphically  portray  a  complex  decision  analysis  problem  with 
multiple  conflicting  criteria  and  alternatives  in  much  the 
same  fashion  as  in  figure  1.  Once  the  attributes  are 
modeled,  the  decision-maker  can  make  both  objective  and 
subjective  judgments  about  the  factors  and  criteria,  through 
which  alternatives  are  ranked  by  the  program  in  a  manner 
which  is  consistent  with  these  judgments. 

The  judgments  are  entered  in  one  of  two  ways: 
objectively  or  subjectively.  In  the  objective  fashion,  the 
user  can  enter  numerical  scores  for  the  different  factors. 
Subjectively,  the  user  is  presented  with  the  criteria  as  a 
series  of  pairs  and  then  asked  to  make  subjective  statements 
about  the  pairs.  These  statements  are  of  the  nature  of 
whether  or  not  one  alternative  is  better  than  another  with 
respect  to  this  criteria.  The  user  can  make  judgments 
ranging  from  "they  are  equal"  up  through  "alternative  one  is 
extremely  preferable  to  alternative  two"  and  so  on.  A 
typical  model,  as  created  by  the  members  of  the  experimental 
group  can  be  found  in  Appendix  B. 

The  program  then  synthesizes  the  information  that  is 
input  in  a  consistent  manner  and  provides  a  rank-ordered 
list  of  alternatives,  given  that  information  furnished  by 
the  user.  The  decision-maker  can  then  perform  sensitivity 
analysis  on  the  decision  and  the  criteria  in  order  to  fully 
understand  the  decision-making  process.  Expert  Choice  also 
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reports  on  the  logical  consistency  of  the  decision  maker  in 
the  judgments  that  have  been  made  and  allows  for  corrections 
in  this  area. 

Expert  Choice  is  an  excellent  adaptation  of  the  AHP  for 
the  computer.  It  was  chosen  for  this  research  because  of 
its  theoretical  foundation  in  AHP  and  its  ease  of  use. 
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Ill .  Methodology 


"Research  Design,"  "Discussion  of  Variables,"  and 
"Discussion  of  Hypothesis"  entail  the  three  major  areas  of 
methodology  covered  in  this  chapter.  "Research  Design," 
details  the  experimental  design  that  will  test  the  null  and 
research  hypotheses.  "Discussion  of  Variables,"  outlines 
the  major  constructs,  the  pertinent  variables  and  the  null 
and  research  hypotheses  of  this  study.  This  section 
concludes  with  a  discussion  of  the  operationalization  of  the 
variables.  "Discussion  of  Hypotheses,"  lists  the  null  and 
research  hypotheses  to  be  tested  by  the  research. 

Research  Design 

A  post-test-only  control  group  design  was  used  to  test 
the  hypotheses  of  the  study.  The  method  employs  one 
experimental  group  and  one  control  group  as  diagrammed  below 
where  Xy  is  the  experimental  treatment  and  0y/0z  are  post¬ 
test  observations. 


(R)  X,  0, 
(R)  02 
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Arguably,  the  more  common  research  design  might  have 


been  the  pre-test-post-test  control  group  experimental 
design  as  diagrammed  below: 

(R)  0,  X,  03 

(R)  02  04 


The  pretest  notion  is  used  primarily  to  ensure  the 
equality  of  the  two  groups  (Campbell  and  Stanley,  1963, 
p. 25) .  Also,  the  pre-test  can  introduce  a  bias  into  the 
control  group  that  may  invalidate  the  experimental  results 
(Campbell  and  Stanley,  1963,  p.14). 

The  validity  of  the  post-test-only  experimental  design 
is  defended  by  Campbell  and  Stanley  when  they  write: 

while  the  pretest  is  a  concept  deeply 
embedded  in  the  thinking  of  research  workers  in 
education  and  psychology,  it  is  not  actually 
essential  to  true  experimental  designs.  For 
psychological  reasons  it  is  difficult  to  give  up 
"knowing  for  sure"  that  the  experimental  and 
control  groups  were  "equal"  before  the 
differential  experimental  treatment.  Nonetheless, 
the  most  adequate  all-purpose  assurance  of  lack  of 
initial  biases  between  groups  is  randomization. 

Within  the  limits  of  confidence  stated  by  the 
tests  of  significance,  randomization  can  suffice 
without  the  pretest  (Campbell  and  Stanley,  1963, 
p. 25)  . 

Thus,  for  the  above  reasons,  the  post-test-only  control 
group  experimental  design  was  chosen  for  this  research. 
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The  experimental  treatment  in  this  research  is  the 
introduction  of  the  decision  support  system  using  Expert 
Choice  software.  The  control  group  was  asked  to  make  a 
decision  by  any  means  that  they  were  most  comfortable  with. 
As  indicated  by  the  design,  the  assignment  to  the  groups  was 
random  to  control  for  all  variables  except  in  the 
independent  variable  of  interest. 

The  two  groups  were  given  a  briefing  on  the  decision 
scenario  with  multiple  competing  criteria  and  choices.  The 
experimental  group  then  received  the  treatment.  An 
explanation  of  the  Analytical  Hierarchy  Process  and  Expert 
Choice  was  given  to  the  individuals.  The  post-test 
consisted  of  a  complex  decision  scenario.  The  experimental 
group  used  the  DSS,  each  subject  was  given  an  identical 
computer  on  which  the  Expert  Choice  software  was  loaded. 

They  created  their  own  models  and  used  the  DSS  to  aid  in  the 
process.  The  control  group  used  whatever  decision-making 
method  that  they  usually  employ. 

After  completing  the  scenario,  each  subject  filled  out 
a  questionnaire  designed  to  test  the  variables  described 
below  in  the  section  entitled  "Discussion  of  the  Variables 
and  Major  Hypotheses."  Univariate  and  bivariate  analyses 
were  performed  on  the  data  to  draw  conclusions  and 
recommendations  from  the  research. 
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This  experimental  design  controls  the  threats  to 
internal  validity  quite  well  (Emory,  1985:121).  If  there  is 
a  problem  with  internal  validity  it  is  in  separating  the 
effects  of  the  AHP  and  the  DSS.  However  since  the  DSS  is 
largely  based  on  the  AHP  this  may  not  be  necessary.  The 
goal  of  this  research  is,  after  all,  to  search  for  an 
effective  decision-making  aid.  If  the  combined  effects  of 
the  AHP  and  the  DSS  are  positive  then  the  individual  roles 
of  each  are  inconsequential .  The  individual  roles  of  each 
may  be  the  subject  of  further  research. 

The  post-test-only  control  group  experimental  design  is 
not  as  strong  in  the  field  of  external  validity.  There  is 
the  possibility  that  subjects  were  affected  by  the  act  of 
being  tested;  however,  this  is  more  of  a  problem  in  studies 
in  which  the  pretest  introduces  unusual  activities  (Emory, 
1985:121).  Since  the  pre-test  has  been  eliminated  and  the 
experiment  was  conducted  with  AFIT  students  familiar  with 
the  acquisition  process,  this  may  not  be  a  problem. 

A  second  problem  with  external  validity  is  whether  or 
not  the  findings  are  generalizable  to  the  PCOs,  PMs,  and 
SSAs  of  the  acquisition  community;  45  AFIT  students  may  not 
be  a  representative  sample.  A  subjective  case  may  be  made, 
however,  that  these  subjects  were  relatively  typical  if 
program  managers,  contracting  officers  and  source  selection 
authorities.  This  generalization  is  based  on  the  similar 


27 


education,  training  and  career  paths  of  the  population 
studied,  compared  with  that  which  apparently  exists  in  the 
acquisition  community. 

Discussion  of  Variables 

Major  Constructs.  The  following  constructs  are 
contained  in  the  research  design  and  warrant  definition: 

1.  Decision:  A  solution  that  ends  uncertainty  or 

dispute. 

2.  Decision-Making  Process:  The  method  by  which 
the  decision  maker  chooses  among  competing  alternatives  to 
achieve  the  preferred  state  and  end  the  uncertainty  or 
dispute. 

3.  The  Analytical  Hierarchy  Process:  A  process 
through  which  a  complex  situation  is  broken  down  into  its 
components.  These  components  are  arranged  into  a  hierarchic 
order  according  to  importance.  Subjective  judgments  are 
assigned  numerical  values  according  to  their  relative 
importance.  Finally,  the  judgments  are  synthesized  to 
determine  which  components  should  hold  priority  in  the 
overall  situation. 

Variables .  The  variables  used  in  this  design  were 
derived  from  the  literature  review  and  the  author's 
judgement  of  the  factors  important  to  test  the  hypotheses. 
Demographic  variables  were  included  to  aid  in  describing  the 
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population  and  the  sample  used  to  test  the  hypotheses.  The 
demographic  variables  were  also  used  to  explain  the 
relationships  between  the  sample  and  the  results  achieved  in 
the  testing  of  the  dependent  variables.  Finally,  the 
demographic  variables  were  used  to  verify  the  lack  of  bias 
in  the  two  groups. 

Independent  Variable.  Xi — The  use  of  the  Expert 
Choice  Software  model  based  on  the  Analytical  Hierarchy 
Process.  The  subjects  were  given  an  introduction  to  Expert 
Choice  and  its  modeling  capabilities.  Included  in  this 
introduction  was  a  description  of  the  Analytical  Hierarchy 
Process.  Subjects  were  then  given  a  complex  decision 
scenario  and  asked  to  make  the  best  possible  decision  using 
the  DSS.  Each  subject  was  assigned  to  an  identical  computer 
onto  which  Expert  Choice  had  been  loaded. 

Dependent  Variables.  The  following  are  the 
dependent  variables  used  to  test  the  hypotheses. 

1.  Effectiveness.  This  is  a  measurement  of  the 
subjects  ability  to  make  the  "best"  decision.  This  "best'' 
decision  was  determined  by  a  "panel  of  experts"  who  were 
given  the  decision  scenario  generated  by  the  author  in 
advance  of  the  experiment.  The  panel  used  the  guidance 
available  and  any  method  that  they  wished  to  employ  to  solve 
the  problem.  The  subjects'  decisions  were  then  compared  to 
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the  decision  of  the  panel.  If  they  agreed,  then  the  subject 
made  an  "effective"  decision. 

2.  Consistency.  Consistency  was  measured  by  the 
number  of  the  individuals  in  the  control  group  and  the 
experimental  group  who  chose  the  same  outcome.  This 
variable  is  important  to  avoid  the  appearance  that  the 
decision-making  process  is  arbitrary. 

3.  Speed.  The  actual  amount  of  time  taken  to 
make  the  decision.  All  subjects  were  timed  during  their 
decision-making  process. 

4.  Difficulty.  Ease  of  the  process.  This  was 
measured  by  a  post-test  survey  and  based  on  the  feelings  of 
the  subjects. 

5.  Understanding.  How  well  does  one  comprehend 
the  process  used  to  make  the  decision?  This  was  measured  by 
administering  a  post-test  survey  and  examining  the  answers 
of  the  subjects. 

6.  Confidence.  A  measure  of  the  decision-maker's 
feeling  that  the  alternative  chosen  was  correct. 

Demographic  Variables.  The  following  demographic 
variables  served  as  control  variables  to  analyze  their 
relationship  to  the  dependent  variable  and  to  analyze  their 
effect  on  the  relationship  between  the  dependent  and  the 
independent  variables. 


30 


1.  Education.  This  variable  records  the  highest 
level  of  education  achieved  by  the  subject.  The  scale 
ranges  from  "High  School"  through  "Post-Graduate  or 
Professional  Degree." 

2.  Major  Field  of  Study.  If  the  subject  has 
completed  sufficient  education  to  have  a  concentration,  this 
variable  records  that  area. 

3.  Formal  acquisition  related  courses.  This 
variable  records  the  familiarity  with  the  acquisition 
process  by  the  measure  of  formal  education  in  the  field. 

4.  Courses  in  field  of  expertise.  This  variable 
measures  the  formal  education  received  in  the  subjects' 
specific  field  of  expertise  in  the  work  place. 

5.  Experience.  Subjects  were  asked  for  the 
number  of  years  of  specific  experience  they  have  had  in 
their  field  of  expertise. 

6.  Certification.  Subjects  were  asked  if  they 
have  been  certified  in  their  field  by  a  professional 
organization  to  further  measure  familiarity  in  the  field. 

Other  Variables  of  Interest.  The  following 
questions  were  asked  to  examine  the  subjects'  feelings  about 
the  decision-making  process  and  the  specific  scenario  at 
hand. 

1.  If  you  had  more  time,  could  you  have  made  a 
better  decision?  Did  the  subjects  feel  hurried? 
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2.  Would  you  classify  this  decision  as  structured 
(programmed)  or  unstructured  (non-programmed) ?  Because  the 
use  of  a  Decision  Support  System  lends  itself  more  readily 
to  a  less  structured  problem,  this  question  measured  the 
subject's  view  of  the  problem  at  hand.  If  the  acquisition 
process  upon  which  the  experiment  is  based  is  a  highly 
structured  process,  then  one  would  expect  little  difference 
between  the  groups  on  the  dependent  variables. 

3.  How  did  you  feel  about  this  process?  The 
subjects  were  given  a  scale  ranging  from  "Frustrated"  to 
"Joy"  in  order  to  get  a  feel  for  their  comfort  in  tne 
situation. 

Operationalization  of  the  Variables. 
Operationalization  of  the  variables  was  accomplished  by  a 
written  questionnaire  that  was  administered  after  the 
experiment.  To  briefly  recapitulate,  the  independent 
variables  were:  effectiveness,  consistency,  speed, 
difficulty,  understanding,  and  confidence.  The  first  three 
(effectiveness,  consistency,  and  speed)  lend  themselves  well 
to  an  objective  measurement.  While  the  last  three 
(difficulty,  understanding,  and  confidence)  required  the 
subjects  to  respond  to  statements  relating  the  extent  to 
which  they  agreed  or  disagreed  with  statements  about  the 
process. 
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An  objective  measure  can  be  used  to  measure 


effectiveness,  speed,  and  consistency.  Effectiveness  was 
defined  above  as  the  number  of  "correct"  answers.  This  was 
measured  counting  the  number  of  answers  that  corresponded  to 
the  answer  provided  by  the  "panel  of  experts."  Consistency 
is  the  frequency  of  the  same  answer.  This,  again  was 
measured  objectively.  Finally  speed  was  simply  measured  '  y 
the  elapsed  time  required  to  make  the  decision. 

The  last  three  dependent  variables  were  measured  by 
asking  the  subjects  to  express  agreement  or  disagreement 
with  statements  relating  to  the  variables.  The  Likert-type 
scale  below  was  used  to  prompt  the  subject  responses. 

STRONGLY  MILDLY  3  NEUTRAL  MILDLY  STRONGLY 

1  AGREE  2  AGREE  4  DISAGREE  5  DISAGREE 


The  Likert-type  scale  is  frequently  used  because  of  its 
advantages  for  statistical  uses.  These  scales  are  more 
suited  to  statistical  analysis  than  free-form  questions. 

The  Likert-type  scale  is  verified  by  Kidder  as  being  the 
most  frequently  used  scale  to  analyze  attitudes  (Kidder, 
1981,  p.215).  The  answers  provided  by  the  subjects  are 
represented  by  numerical  scores  in  this  system  and  can  be 
subjected  to  numerical  analysis. 

Where  indicated  by  initial  investigation,  a 
correlational  analysis  was  performed  on  the  variables  of 
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interest.  Because  the  data  was  condensed  into  an  ordinal 

form,  Gamma  was  used  as  a  correlation  coefficient.  Gamma, 

as  with  most  correlational  coefficients,  ranges  from  -1.0  to 

+1.0.  Meier  and  Brudney  describe  Gamma  as  follows: 

What  gamma  (and  many  other  ordinal  measures  of 
association)  does  is  take  the  difference  between 
the  number  of  concordant  or  consistently  ordered 
pairs  and  the  number  of  discordant  or 
inconsistently  ordered  pairs  in  the  cross¬ 
tabulation.  This  difference  indicates  the 
relative  support  in  the  contingency  table  for  a 
positive  as  opposed  to  a  negative  relationship 
between  the  two  variables.  If  the  number  of 
concordant  pairs  exceeds  the  number  of  discordant 
pairs,  then  on  balance,  there  is  greater  support 
for  a  positive  relationship  in  the  table.  In  that 
case  the  difference  between  them  will  be  positive, 
and  the  gamma  statistic  will  have  this  sign.  On 
the  other  hand,  if  the  number  of  concordant  pairs 
is  less  than  the  number  of  discordant  pairs,  there 
is  greater  support  for  a  negative  relationship. 

The  difference  between  them  will  be  negative,  and 
this  will  be  reflected  in  the  (negative)  sign  of 
gamma.  Regardless  of  the  direction  of  the 
relationship,  the  larger  the  difference  between 
the  number  of  concordant  and  the  number  of 
discordant  pairs,  the  greater  is  the  association 
between  the  two  variables,  and  the  greater  is  the 
magnitude  of  gamma  (Meier  and  Brudney,  1981:232). 


The  calculation  for  gamma  is  shown  below. 


#  of  concordant  pairs  -  #  of  discordant  pairs 
Gamma  =  _ 


#  of  concordant  pairs  +  #  of  discordant  pairs 
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Operationalization  of  the  demographic  variables  and  the 
variables  termed  "other  variables  of  interest"  was  achieved 
by  using  open-ended  questions.  The  different  demographic 
variables  were  measured  with  responses  to  alternatives. 
Wherever  possible,  the  response  of  "other"  was  provided  with 
the  prompt  to  provide  the  information  free-form.  The 
variables  described  as  "other  variables  of  interest"  were 
open  ended  questions  that  allowed  the  subject  to  respond 
free-form.  These  were  analyzed  for  recurring  trends. 

For  specifics  on  the  questionnaire,  see  the  copy  found 
in  the  appendix. 

Discussion  of  the  Hypotheses 

In  chapters  one  and  two,  the  nature  and  complexity  of 
the  defense  acquisition  process  were  discussed.  Given  the 
human  inclination  to  satisfice,  the  author  suspected  that 
the  best  possible  decisions  were  not  being  made  by  the 
parties  in  the  process.  As  discussed  in  chapter  two  (Simon, 
Lindblom,  and  others) ,  the  limited  capability  of  the  human 
mind  is  a  major  circumstance  that  excludes  optimizing  in 
complex  decision  scenarios. 

Saaty  (also  discussed  in  chapter  two)  proposes  the 
Analytical  Hierarchy  Process  as  a  method  to  increase  the 
ability  to  optimize.  The  use  of  the  AHP,  coupled  with  the 
aid  of  computer  aided  decision  apport  systems  should  aid  in 
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the  decision-making  process.  From  this  the  author  suspects 
that  the  process  can  be  improved  through  the  use  of  these 
tools. 

In  this  research,  variables  have  been  tested  to  measure 
the  degree  to  which  the  decision-making  process  is  improved 
by  the  use  of  a  computer  aided  decision  support  system, 
specifically  Expert  Choice.  The  objective  of  the  research 
is  to  identify  the  variables,  if  any,  that  are  affected  by 
the  DSS. 

To  accomplish  this  objective  the  following  null  and 
research  hypotheses  were  formulated: 

Null  Hypotheses.  The  following  is  a  discussion  of  the 
null  hypotheses. 

H^:  The  use  of  a  DSS  will  have  no  effect  on  the 

effectiveness  of  the  decision-making  process. 

Hq2:  The  use  of  a  DSS  will  have  no  effect  on  the 

consistency  of  the  decision-making  process. 

Ho3:  The  use  of  a  DSS  will  have  no  effect  on  the  speed 

of  the  decision-making  process. 

Ho*:  The  use  of  a  DSS  will  have  no  effect  on  the 

difficulty  of  the  decision-making  process. 

H05:  The  use  of  a  DSS  will  have  no  effect  on  the 

understanding  of  the  decision-making  process  for  the 
participants . 
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H^:  The  use  of  a  DSS  will  have  no  effect  on  the 

decision  maker's  confidence  in  his  decision. 

Research  Hypotheses.  The  following  is  a  discussion  of 
the  research  hypotheses. 

Hi:  The  use  of  a  DSS  will  increase  the  effectiveness 

of  the  decision-making  process. 

H2:  The  use  of  a  DSS  will  increase  the  consistency  of 

the  decision-making  process. 

H3:  The  use  of  a  DSS  will  increase  the  speed  of  the 

decision-making  process. 

H4:  The  use  of  a  DSS  will  make  the  decision-making 

process  less  difficult. 

H5:  The  use  of  a  DSS  will  increase  the  understanding 

of  the  decision-making  process  for  the  participants. 

H6:  The  use  of  a  DSS  will  increase  the  decision 

maker's  confidence  in  his  decision. 
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This  chapter  will  focus  primarily  on:  1)  Demographics, 


2)  Findings,  3)  Hypotheses,  and  4)  Free-Form  Data.  In  the 
section  entitled  demographics,  the  general  make-up  of  the 
control  and  experimental  group  will  be  discussed.  The 
section  on  findings  will  address  the  validity  of  the  null 
and  research  hypotheses  proposed  in  the  previous  chapter. 
The  section  on  hypotheses  will  address  the  implications  of 
the  findings  on  the  research  hypotheses  proposed  in  the 
section  entitled  "Methodology.”  Finally,  the  section  on 
free-form  data  will  discuss  the  feelings  of  the  subjects 
based  on  open  ended  questions  in  the  questionnaire. 

Demographics 

Demographic  information  was  gathered  through  a  series 
of  questions  on  the  questionnaire.  This  section  will 
address  each  question  and  participant  responses.  For 
details  on  the  questionnaire,  see  the  copy  found  in  the 
appendix.  The  statistics  are  reported  in  percentages. 

The  following  five  questions  were  used  to  verify  the 
level  of  expertise  of  the  subjects  and  to  ensure  that  there 
was  no  bias  due  to  a  difference  in  expertise  between  the 
experimental  and  control  groups. 
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Question  1:  What  is  the  highest  level  of  education  that  you 
have  completed? 

The  following  results  are  summarized  in  figure  2. 


Control  Group:  Of  the  control  group  96%  reported  some 
graduate  level  education  with  4%  reported  already  having 
achieved  a  graduate  level  degree. 

39 


Experimental  group:  Some  graduate  level  education  was 
reported  by  95%,  while  the  remaining  5%  reported  a  graduate 
degree . 

No  essential  difference  exists  between  the  groups.  The 
high  level  of  education  achieved  by  the  groups  is  not 
surprising  since  the  experiment  was  conducted  on  graduate 
students  at  the  Air  Force  Institute  of  Technology. 

Question  2:  What  was  your  major  field  or  area  of  study? 


Control  group:  "Business"  was  reported  by  19%  as  their 
major  field  of  study  while  12%  reported  engineering,  but 
the  clear  majority  reported  some  form  of  systems  management. 
Experimental  group:  Of  this  group,  11%  reported  "Business," 
53%  "Engineering"  and  37%  reported  some  form  of  systems 
management.  The  results  of  the  experimental  group  indicate 
that  although  the  entire  group  is  currently  studying  systems 
management  at  AFIT,  group  members  identify  more  readily  with 
engineering.  While  there  appears  to  be  a  difference  here, 
the  free  form  questions  indicated  that  there  is  almost  no 
difference  in  the  major  field  between  the  two  groups. 
Question  3:  How  many  acquisition  related  courses  have  you 
taken? 

Control  group:  Figure  4  shows  that  19%  reported  having 
taken  no  courses,  54%  had  taken  one  to  three,  23%  four  to 
six,  4%  seven  to  ten,  while  no  individuals  reported  more 
than  ten. 

Experimental  group:  In  this  group,  11%  reported  no  courses 
taken,  37%  reported  having  taken  one  to  three,  47%  four  to 
six,  and  5%  seven  to  ten.  Again,  no  subjects  reported  more 
than  ten  courses.  These  results  indicate  a  good  solid  level 
of  training  in  the  acquisition  field.  Both  groups  were 
centered  around  the  "one  to  three"  area  and  the  "four  to 
six"  area.  If  there  is  any  difference,  the  experimental 
group  seems  to  have  had  somewhat  more  training  in  the 
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acquisition  field.  This  difference  could  suggest  an  area 
for  further  research;  however,  later  results  will  indicate 
that  there  was  virtually  no  difference  between  groups  on  the 
choice  of  contractor.  This  lack  of  difference  would 
indicate  that  this  difference  in  training  had  little  or  no 
effect  on  the  outcome  of  the  experiment. 

ACQUISITION  RELATED  COURSE 

60% 

60% 

40% 

30% 

20% 

10% 

0% 

Figure  4  Distribution  of  the  Number  of  Courses  Taken 

Question  4:  How  many  courses  have  you  taken  in  your  field 
of  expertise? 

Control  group:  Here,  19%  reported  no  formal  courses  in 
their  field  of  expertise,  50%  reported  between  one  and  four 
courses.  The  remaining  31%  reported  five  or  more. 
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Experimental  group:  Only  5%  reported  having  taken  no 
courses,  but  53%  reported  between  one  and  four;  42%  reported 
over  five  courses  completed  in  their  field  of  expertise. 


COURSES  IN  FIELD 


ES  CONTROL  SSI  EXPERIMENTAL 


Figure  5  Distribution  of  the  Number  of  Courses  Taken 


Again,  there  is  a  slight  difference  in  the  educational  level 
in  field  towards  the  experimental  group,  however,  the 
difference  is  so  small  as  to  be  inconsequential.  The  same 
argument  applies  here  as  above  due  to  the  lack  of  difference 
in  the  choice  of  the  contractors.  This  small  difference  in 
education  could  have  little  effect  on  the  outcome  of  the 
experiment. 
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Question  5:  How  many  years  of  experience  do  you  have  in 
your  field  of  expertise? 

Control  group:  Figure  6  shows  that  35%  reported  having  less 
than  1  year  of  experience  in  their  current  field,  30% 
reported  between  one  and  four  years  while  the  remaining  35% 
reported  five  or  more  years  of  experience. 

Experimental  group:  Here,  26%  reported  having  no  experience 
in  their  current  field,  58%  reported  between  one  and  four 
years  of  experience  while  only  16%  reported  five  or  more 
years. 

The  control  grouj  holds  the  edge  in  the  years  of 
practical  experience  with  35%  reporting  five  or  more  years. 

YEARS  EXPERIENCE 
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B9  CONTROL  S3  EXPERIMENTAL 
Figure  6  Distribution  of  Experience  in  Years 
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Question  6:  Are  you  certified  in  your  field? 

Control  group:  Only  8%  reported  certification  as  some  level 
of  program  manager.  All  other  responses  indicated  no 
certification. 

Experimental  group:  Certification  of  program  manager  was 
reported  by  11%.  Again,  all  other  responses  were  "not 
certified. " 
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Figure  7  Distribution  of  Certified  Subjects 


CERTIFICATION 
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The  data  from  this  question  does  not  indicate  any 
concern  of  differences,  nor  is  it  surprising  due  to  the 
level  of  the  subjects  in  their  career  field.  As  figure  7 
indicates,  the  results  on  this  question  are  almost 
identical . 

When  using  a  post-test  only  control  group  experimental 
design,  a  major  concern  is  that  of  bias  in  the  groups 
(Campbell  and  Stanley,  1963,  p.25).  However,  the  results  of 
the  demographic  questions  indicate  that  randomization  has 
achieved  its  purpose.  As  Campbell  and  Stanley  state,  "the 
most  adequate  all-purpose  assurance  of  lack  of  initial  bias 
is  randomization"  (Campbell  and  Stanley,  1963,  p.25).  The 
above  analysis  clearly  indicates  that  the  two  groups  do  not 
contain  any  significant  biases  that  would  invalidate  the 
results  of  the  experiment. 

Findings 

These  findings  were  based  on  the  research  questions 
found  in  the  questionnaire.  As  discussed  in  the  chapter 
entitled  "Methodology,"  the  variables  "effectiveness," 
"consistency,"  "speed,"  "difficulty,"  "understanding,"  and 
"confidence"  were  used  to  test  the  research  hypotheses. 

Each  question  was  examined  individually  and  the  results 
were  summarized.  Overall  conclusions  indicated  by  the 
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results  are  reserved  for  the  final  chapter  entitled 


"Recommendations  and  Conclusions." 

Effectiveness .  This  variable  was  measured  objectively 
by  comparing  the  number  of  "correct"  answers  in  the  groups. 
In  each  group  85%  of  the  subjects  chose  the  "correct" 
answer —  Magnetic  Technologies  Inc.  Only  8%  of  the  control 
group  chose  Fusetech,  4%  chose  Smith  and  Jones,  and  4%  chose 
Techco.  The  experimental  group  had  remarkably  similar 
results.  Here  10%  chose  Fusetech  5%  chose  Techco.  Figure  8 
shown  below  indicates  very  little  difference  between  the 
groups . 


CHOICE  OF  CONTRACTOR 


BS  CONTROL  ES3  EXPERIMENTAL 


Figure  8  Distribution  of  the  Choice  of  Contractor 
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Consistency.  This  variable  was  also  measured 
objectively  by  comparing  the  distribution  of  the  answers. 
Figure  8,  again,  shows  that  the  two  groups  chose  the  same 
alternative  with  consistency. 

Speed.  By  timing  the  decision-making  process,  an 
objective  measure  of  speed  was  obtained. 

TIME  IN  MINUTES 
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Figure  9  Distribution  of  Time  in  Minutes 
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Figure  10  shows  that  the  control  group  tended  to  make 


quicker  decisions  than  the  experimental  group.  This  was,  at 
first,  a  great  surprise  to  the  author;  however,  some  of  the 
answers  to  the  free-form  questions  yielded  an  indication 
that  the  control  group  may  have  satisficed.  These  free-form 
questions  are  discussed  later. 

Difficulty,  The  statement  "I  found  the  decision-making 
process  easy"  was  presented  to  the  subjects. 


Figure  10  Distribution  of  the  Level  of  Difficulty 
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Figure  10  above  indicates  a  difference  between  the  groups. 
While  only  8%  of  the  control  group  and  5%  of  the 
experimental  group  indicated  any  disagreement  with  the 
statement  at  all,  experimental  group  members  were  much 
stronger  in  their  feelings.  This  degree  of  feeling  was 
indicated  by  the  number  of  "strongly  agree"  answers.  The 
section  below  entitled  "Hypotheses"  contains  a  correlational 
analysis  investigating  this  difference.  While  it  was  not 
surprising  that  the  experimental  group  found  the  process  to 
be  easy,  it  was  surprising  that  the  control  group  found  it 
so.  Again,  this  may  indicate  that  the  control  group 
employed  some  form  of  satisficing.  This  postulate  was 
supported  by  the  free-form  questions  and  will  be  discussed 
later. 

Understanding .  Figure  11  represents  understanding. 

This  variable  was  measured  through  the  use  of  a  Likert- 
type  scale  attached  to  the  following  statement  "I  understood 
the  process  by  which  the  contractor  was  chosen."  The 
majority  of  the  respondents  in  both  groups  were  bunched  at 
the  responses  labeled  "Strongly  Agree"  and  "Mildly  Agree"  as 
seen  above  in  figure  12.  There  were  significantly  more 
"Strongly  Agree"  answers  in  the  experimental  group  which 
indicated  an  area  for  further  study.  A  correlational 
analysis  was  performed  and  the  results  are  reported  below  in 
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UNDERSTANDING 


ggS  CONTROL  ESS3  EXPERIMENTAL 


Figure  11  Distribution  of  the  Level  of  Understanding 


the  section  entitled  "Hypotheses . "  There  was  one  "Strongly 
Disagree"  in  the  experimental  group,  but  this  seemed  to  be 
the  result  of  not  understanding  the  computer. 

Confidence.  This  variable  was  also  measured  by  a 
Likert-type  scale.  The  statement  examined  was  "I  am 
confident  that  I  made  the  right  decision."  As  seen  in 
figure  12,  there  may  be  a  slight  difference  in  the  groups 
indicating  a  small  increase  in  confidence  for  the 
experimental  group.  This  finding  was  supported  by  a 
correlational  analysis  shown  in  a  later  section. 


51 


CONFIDENCE 


50% 

40% 

30% 

20% 

10% 

0% 

SS9  CONTROL  EXPERIMENTAL 

Figure  12  Confidence 

There  was  one  interesting  finding  in  this  area  discovered  in 
the  free-forx.  "swers.  A  single  respondent  from  the 
experimental  group  reported  the  computer  indicating  the 
correct  decision,  yet  the  subject  refused  to  trust  the 
model.  The  subject  changed  his  decision  to  an  incorrect 
answer  based  on  a  lack  of  confidence  in  the  model. 
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Hypotheses 

The  following  are  of  the  implications  of  the  above 
findings  as  they  relate  to  the  null  and  research  hypotheses 
proposed  in  the  methodology  section.  Each  hypothesis  will 
be  discussed  individually.  An  overall  discussion  of  the 
findings  can  be  found  in  the  final  chapter. 

Hoi:  The  use  of  a  DSS  will  have  no  effect  on  the 

effectiveness  of  the  decision-making  process. 

This  null  hypothesis  cannot  be  rejected.  An 
examination  of  the  results  to  the  effectiveness  measure 
reveals  that  there  is  little  or  no  evidence  to  support  that 
the  experimental  group  experienced  any  increase  in 
effectiveness  due  to  the  DSS.  The  percentage  of  correct 
answers  between  the  two  groups  was  identical. 

Hi!  The  use  of  a  DSS  will  increase  the  effectiveness 
of  the  decision-making  process. 

This  research  hypothesis  must  be  rejected.  There  was 
no  difference  between  the  groups. 

Ho2:  The  use  of  a  DSS  will  have  no  effect  on  the 

consistency  of  the  decision-making  process. 

Again,  there  is  no  evidence  to  reject  this  null 
hypothesis.  The  two  groups  showed  an  almost  identical 
distribution  for  consistency  purposes.  The  only 
inconsistency  was  that  the  control  group  had  one  individual 
choose  Smith  &  Jones. 
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H2:  The  use  of  a  DSS  will  increase  the  effectiveness 

of  the  decision-making  process. 

Again,  this  research  hypothesis  cannot  be  supported  by 
the  results  obtained.  There  was  no  difference  between  the 
two  groups  in  consistency. 

Ho3:  The  use  of  a  DSS  will  have  no  effect  on  the  speed 

of  the  decision-making  process. 

The  results  of  the  experiment  in  figure  9  seem  to 
justify  the  opposite  of  what  was  expected.  Certainly,  this 
null  hypothesis  must  be  rejected.  The  control  group  worked 
much  faster  than  the  experimental  group.  Some  of  this 
discrepancy  can  be  explained  because  the  experimental  group 
was  relatively  unfamiliar  with  the  software,  having  received 
a  one-hour  indoctrination  prior  to  the  experiment.  Most  of 
the  subjects  indicated  that  if  they  had  been  more  familiar 
with  the  software,  they  would  have  worked  more  quickly. 

This  factor  alone  may  be  insufficient  to  explain  the  large 
difference  between  groups. 

H3:  The  use  of  a  DSS  will  increase  the  speed  of  the 

decision-making  process. 

This  research  hypothesis  was  completely  negated  by  the 
findings.  The  experimental  group,  as  seen  in  figure  9,  was 
clearly  less  speedy  than  the  control  group. 

He*:  The  use  of  a  DSS  will  have  no  effect  on  the  level 

of  difficulty  of  the  decision-making  process. 
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The  results  indicate  that  the  experimental  group  felt 
more  strongly  about  the  ease  of  the  process  as  indicated  by 
the  high  percentage  of  "strongly  agree"  responses  to  the 
"difficulty"  question.  The  null  hypothesis  is  therefor 
rejected.  A  correlational  analysis  was  performed  on  the 
variable  "difficulty"  and  the  use  of  the  DSS.  Figure  13 
summarizes  the  results. 
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Figure  13  Cross-Tabulation  of  DSS  and  Difficulty 
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As  seen,  the  Gamma  value  of  -.58  indicates  moderate  to 
strong  correlation  between  the  use  of  the  DSS  and  how 
difficult  a  subject  found  the  process  to  be.  The  cross¬ 
tabulation  shows  that  of  those  people  responding  who  agreed 
that  the  process  was  easy,  64.3%  were  in  the  experimental 
group.  Only  32.3%  of  the  subjects  who  disagreed  were  in  the 
experimental  group,  for  a  difference  of  32.1%.  Clearly,  the 
null  hypothesis  must  be  rejected —  a  large  difference  exists 
between  the  two  groups.  This  becomes  important  in  light  of 
the  evidence  that  the  decisions  were  accurate  with  or 
without  the  DSS.  Certainly,  is  suspect  for  rejection. 

H The  use  of  the  DSS  will  make  the  decision-making 
process  less  difficult. 

As  discussed  above,  there  is  clear  evidence  to  accept 
this  hypothesis.  The  analysis  shows  that  the  experimental 
group  found  the  process  to  be  less  difficult. 

Ho5:  The  use  of  a  DSS  will  have  no  effect  on  the  level 

of  understanding  of  the  decision-making  process  for  the 
participants. 

Nearly  all  of  the  respondents  in  both  the  control  group 
and  the  experimental  group  reported  a  level  of  understanding 
the  process;  however,  the  experimental  group  indicated  a 
stronger  feeling.  A  correlational  analysis  was  performed  to 
investigate  this  phenomenon. 
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Figure  14  Cross-Tabulation  of  DSS  and  Understanding 


Although  Gamma  is  only  -.28,  this  still  indicates  a 
weak  to  moderate  correlation  between  the  use  of  the  DSS  and 
the  level  of  understanding  of  the  subject.  However,  there 
is  still  a  clear  difference  between  the  groups.  When  asked 
whether  they  understood  the  process,  65%  of  the  subjects  who 
reported  a  low  understanding  were  in  the  control  group.  Only 
35%  were  in  the  experimental  group. 

A  single  respondent  from  the  experimental  group 
reported  "strong  disagreement"  to  the  understanding  prompt. 
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That  respondent  went  on  to  report  that  the  lack  of 
understanding  was  related  to  the  software,  not  the  decision¬ 
making  process. 

H5:  The  use  of  a  DSS  will  increase  the  understanding 

of  the  decision-making  process  for  the  participants. 

The  difference  between  the  two  groups,  although  less 
pronounced,  is  still  not  to  be  ignored.  These  results 
indicate  that  the  use  of  the  DSS  did  increase  the 
understanding  for  the  participants. 

H^:  The  use  of  a  DSS  will  have  no  effect  on  decision 

maker's  confidence  in  his  decision. 

This  is  also  an  area  in  which  tangible  differences  can 
be  asserted.  The  experimental  group  showed  more  confidence 
in  their  decisions  than  did  the  control  group.  31%  of  the 
control  group  reported  "mild  agreement"  to  the  confidence 
measure,  but  only  27%  were  willing  to  report  strong 
agreement.  Only  58%  report  some  confidence.  In  contrast, 
32%  of  the  experimental  group  reported  "strong  agreement" 
and  74%  reported  some  form  of  agreement. 

Figure  15  shows  the  results  of  a  correlational  analysis 
performed  on  this  variable. 

Figure  15  indicates  that  those  using  the  DSS  were  more 
confident  in  their  decisions  than  were  the  subjects  not 
using  the  DSS.  Hq6  must  be  rejected. 
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Figure  15  Cross-Tabulation  of  DSS  and  Confidence 

H6:  The  use  of  a  DSS  will  increase  the  decision¬ 

maker's  confidence  in  his  decision. 

This  hypothesis  supported  by  the  data.  The  the 
experimental  group  was  more  confident  in  its  decisions. 
Free-Form  Data 

The  following  free-form  questions  were  asked  of  the 
subjects  to  gain  further  insight  into  the  process. 

1.  If  you  had  had  more  time  could  you  have  made  a  "better" 
decision? 
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There  were  only  two  2  of  26  subjects  in  the  control 
group  who  felt  that  they  needed  more  time.  This  is  not 
surprising  given  the  speed  with  which  the  control  group 
completed  the  exercise.  The  experimental  group  yielded  some 
surprising  results.  Although  the  majority  of  subjects  made 
the  correct  decision,  six  out  of  19  felt  that  with  more  time 
they  could  have  made  a  better  decision. 

2.  Would  you  classify  this  decision  as  structured 
(programmed)  or  unstructured  (non-programmed) ? 

Over  90%  of  each  group  felt  that  the  problem  presented 
was  structured.  One  respondent  in  each  group  felt  that  it 
was  both  structured  and  unstructured  and  the  remaining  found 
it  to  be  unstructured.  This  finding  could  explain  the  lack 
of  difference  in  the  effectiveness  between  groups.  The 
review  of  the  literature  reveals  that  Decision  Support 
Systems  are  most  suited  to  semi-structured  to  unstructured 
problems.  Since  the  subjects  found  this  to  be  a  highly 
structured  problem,  one  would  expect  less  of  an  effect  by 
the  DSS  in  the  area  of  effectiveness.  The  chapter  entitled 
" Recommendations  and  Conclusions"  examines  this  thought 
further. 

3.  How  did  you  feel  about  this  process? 

The  distribution  below  describes  the  responses  to  this 
question.  A  majority  of  each  group  reported  either  a 
"relaxed"  feeling  or  a  response  of  "other"  that  indicated  a 
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similar  feeling.  One  subject  on  the  experimental  group  quit 
the  entire  process  early  indicating  extreme  frustration  with 
both  computers  in  general  and  the  software  package.  On  the 
whole,  any  findings  will  not  be  biased  by  feelings  of 
anxiety  within  the  two  groups. 

4.  Why  did  you  choose  this  contractor? 

In  this  section,  the  subjects  were  given  the 
opportunity  to  describe  the  process  by  which  they  made  the 
decision.  The  experimental  group  all  referenced  the 
software  package.  An  examination  of  the  models  constructed 
by  the  group  indicates  that  the  processes  were  almost  all 
identical.  The  control  group  subjects  employed  some  form  of 
satisficing  in  order  to  simplify  the  problem  at  hand. 

The  most  surprising  finding  in  this  area  was  that  all 
of  the  control  group  reported  using  some  variation  of 
Saaty's  Analytical  Hierarchy  Process.  While  the  subjects 
were  unaware  of  the  AHP  by  name,  all  reported  having  split 
the  decision  into  manageable  parts  and  forming  some  form  of 
hierarchy.  While  unexpected,  this  finding  lends  great 
credence  to  Saaty's  claim  that  the  AHP  is  a  good  model  of 
the  human  decision-making  process.  However,  Saaty  would 
undoubtedly  argue  that  the  AHP  is  not  a  satisficing 
approach!  This  will  be  discussed  in  more  detail  later. 
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V.  Conclusions  and  Recommendations 


In  the  conclusion,  a  brief  recapitulation  of  the 
findings  detailed  in  chapter  IV  will  be  presented;  also 
conclusions  will  be  drawn  as  to  the  import  of  those 
findings.  Finally,  brief  suggestions  will  be  made  as  to 
areas  for  further  research  that  were  suggested  by  the 
findings  of  this  thesis. 

Conclusions 

The  results  of  the  experiment  conducted  in  this 
research  were  at  first  surprising  to  the  author.  There  was 
no  increase  in  the  effectiveness  or  consistency  of  the 
decision-makers  of  the  group  that  used  the  DSS.  Theory 
suggests  that  the  human  inclination  to  satisfice  would 
decrease  the  effectiveness  of  those  decision-makers  not 
using  the  DSS.  An  examination  of  the  data  suggests  that  the 
control  group  decision-makers  did  satisfice  in  some  way 
while  simultaneously  employing  some  form  of  AHP.  The 
implications  of  these  results  could  suggest  that  either  1) 
perhaps  it  was  too  simple  to  make  the  ''correct"  decision 
(the  problem  was  actually  more  structured — programmed — than 
it  first  appeared) ,  2)  filtering  the  information  before 
employing  optimization  techniques  is  a  very  effective, 
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albeit  satisficing,  strategy,  or  3)  the  use  of  a  DSS  will 
not  lead  to  "better”  decisions. 

The  possibility  that  the  experiment  was  too  simple  may 
not  be  solvable  at  this  level  of  research.  The  mere  act  of 
writing  the  case  study  lended  a  structure  to  the  problem 
that  may  not  be  there  in  the  real  world.  Theory  suggests 
that  structure  simplifies  the  decision-making  process. 

If  filtering  information  before  using  optimization 
techniques  is  a  very  effective  strategy,  then  that  is  an 
important  finding  in  and  of  itself  and  therefor  warrants 
further  investigation. 

The  possibility  that  the  use  of  a  DSS  may  not  lead  to 

"better"  decisions  was  examined  by  Todd  and  Adams  in  a  paper 

presented  to  the  Ninth  International  Conference  on  Decision 

Support  Systems.  They  challenge  the  notion  that  the  use  of 

a  DSS  will  lead  to  more  effective  decision-making.  In  their 

paper,  they  discuss  the  Prospect  Theory  (Kahneman  and 

Tversky)  and  the  Cost-Benefit  theory. 

The  Prospect  Theory  suggests  that  the  decision-making 

process  does  not  stay  constant  across  subjects  thus 

suggesting  little  or  no  control  by  the  decision-maker.  The 

Prospect  Theory  is  described  as: 

an  example  of  a  perceptual  model  of  decision 
behavior.  It  argues  that,  in  situations  that 
involve  making  risky  decisions,  people  use 
strategies  that  are  inconsistent  with  normative 
models.  Specifically,  Kahneman  and  Tversky 
identify  two  classes  of  aberrations  termed  the 
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certainty  effect  and  the  isolation  effect.  The 
certainty  effect  results  in  decision  makers 
underweighting  outcomes  that  are  simply  probable 
when  compared  to  those  that  are  certain.  The 
certainty  effect  causes  problems  that  are  framed 
as  gains  to  be  approached  in  a  risk  averse 
fashion,  while  those  that  are  framed  as  losses  are 
viewed  in  a  risk  seeking  mode.  The  isolation 
effect  causes  common  characteristics  of  problem 
solutions  to  be  discarded  before  a  selection  is 
made.  Both  of  these  effects  lead  decision  makers 
to  different  solutions  based  upon  the  framing  of  a 
particular  problem  (Todd  and  Adams,  1989:208). 


Todd  and  Adams  go  on  to  conclude: 

The  implication  of  the  Prospect  Theory  is 
that  very  little  progress  is  likely  to  be  made  in 
terms  of  aiding  and  assisting  decision  making.  If 
individuals  do  not  have  conscious  control  over  the 
mechanisms  which  they  use  during  problem  solving, 
then  no  amount  of  training  or  assistance  will 
facilitate  the  improvement  of  decision  making. 

Decision  behavior  may  be  manipulated  in  this  case 
by  a  conscious  framing  of  the  problem  to  invoke 
certain  processes  from  the  decision  maker,  but 
after  that  point,  support  tools  may  be  of  little 
use  (Todd  and  Adams,  1989:209). 

Both  the  AHP  of  Saaty  and  DSS  theory  support  this  idea; 
a  conscious  framing  of  the  problem  will  lead  to  benefits. 

The  Cost-Benefit  Theory  suggests  that  the  decision¬ 
maker  weighs  the  relative  effort  required  to  implement 
decisions  and  then  chooses  an  acceptable  route  with  the 
least  expenditure  of  resources.  Todd  and  Adams  describe  the 
theory: 

Decision  makers  presumably  contrast  the  amount  of 
cognitive  effort  required  to  implement  a 
particular  strategy  with  the  expected  benefits 
associated  with  the  particular  strategy.  The 
benefits  of  the  various  approaches  are  typically 
measured  as  the  likelihood  of  that  approach 
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leading  to  a  good  decision  or  an  accurate  response 
(Payne  1982) .  Given  values  for  cognitive  effort 
and  decision  accuracy,  a  trade-off  is  made.  The 
assumption  is  that,  ideally,  decision  makers  would 
like  to  maximize  the  quality  of  their  decisions 
while  at  the  same  time  minimizing  cognitive 
effort.  However,  to  the  extent  that  these  two 
objectives  are  typically  conflicting,  some  form  of 
compromise  is  required  (Johnson  and  Payne  1985) . 

In  terms  of  DSS  research,  the  key  message  of  Cost- 
Benefit  Theory  is  that  decision  maker  behavior 
cannot  simply  be  viewed  as  being  quality  oriented. 
Thinking  is  hard  and  as  a  result  effort  may  be  an 
important  determinant  of  DSS  use  (Todd  and  Adams, 
1989:209. 


From  this  Todd  and  Adams  suggest: 

If  a  decision  aid  were  to  automate  a  series  of 
strategies,  reducing  the  cognitive  effort  for  each 
but  maintaining  their  relative  degree  of 
difficulty  or  strain,  we  would  anticipate  that  the 
decision  maker  would  continue  to  utilize  the  same 
strategy  as  used  in  the  unaided  environment.  For 
a  decision  aid  to  induce  change  in  this  case,  it 
must  alter  the  effort  rankings  or  relationship 
between  various  strategies.  Consequently,  in  a 
decision  aided  environment,  which  equally  supports 
all  strategies  and  not  changing  the  effort 
relationships  between  them,  there  would  be  no 
strategy  shift  (Todd  and  Adams,  1989:211). 

Todd  and  Adams  conclude  by  suggesting  that  the  thrust 

of  DSS  research  should  be  directed  towards  the  relationships 

between  various  strategies  rather  than  decision  quality 

(Todd  and  Adams,  1989:212).  The  results  of  this  research 

seem  to  support  the  theories  of  Todd  and  Adams:  there  was 

no  increase  in  the  quality  (in  terms  of  effectiveness  and 

consistency)  of  the  decisions  due  to  the  DSS. 
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A  second  important  finding  of  this  research  was  that 
the  use  of  the  DSS  took  more  time  than  unaided  decision¬ 
making.  This  suggests  that  in  the  absence  of  any  positive 
effects,  the  use  of  a  DSS  would  be  a  waste  of  resources. 
However,  the  response  to  some  questions  revealed  that  the 
increased  time  may  be  attributable  to  learning  the  system. 
Also,  the  initial  investment  of  time  required  to  learn  the 
DSS  is  analogous  to  the  "gear  up"  time  in  implementing  any 
new  and  automated  system.  Todd  and  Adams  (as  seen  above) 
postulate  that  the  decision  maker  must  see  a  noticeable 
difference  in  the  effort  expended  in  order  to  make  this 
commitment.  However,  after  an  initial  loss  of  resources, 
the  benefits  should  be  reaped. 

Summarizing  the  positive  effects  of  the  DSS  indicated 
by  the  research,  there  was  a  positive  correlation  between 
the  use  of  the  DSS  and  how  "easy”  the  decision-maker  found 
the  process.  A  positive  correlation  was  also  observed  for 
the  confidence  of  the  decision-maker  in  his  own  decision  and 
with  his  understanding  of  the  decision  making  process  when 
the  DSS  was  used.  All  of  these  results  suggest  areas  for 
further  development  of  the  DSS  concept.  In  the  Air  Force 
environment  where  the  decision-maker  is  constantly  required 
to  explain  and  "sell”  his  decision  to  superiors,  the 
increase  in  confidence  and  understanding  of  the  process 
could  be  invaluable  assets. 
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A  final  intriguing  finding  of  this  research  is  that  all 
of  the  control  group  appeared  to  use  the  Analytical 
Hierarchy  Process  in  some  form  or  another.  This  would,  at 
first  glance,  seem  to  support  Saaty's  contention  that  the 
AHP  is  innate  to  humans  in  the  decision-making  arena. 
However,  further  examination  of  the  data  yielded  that  the 
control  group  used  the  AHP  to  satisfice!  All  of  the 
subjects  used  some  method  to  reduce  or  discount  the 
extraneous  information.  The  AHP  was  designed  as  a  method  to 
induce  optimization  when  making  complex,  unstructured 
decisions.  As  suggested  by  the  review  of  the  literature, 
there  is  a  "bounded  rationality"  exercised  by  the  human  mind 
(Simon  and  Others) .  Perhaps  the  AHP  is  the  innate  model 
used  by  the  human  mind  to  exercise  bounded  rationality  when 
presented  with  overly  complex  scenarios.  Further,  in  less 
complex  situations,  the  "right"  answer  can  apparently  be 
derived  without  the  use  of  computer-assisted  DSS,  even 
though  AHP  is  being  employed. 

Recommendations 

Further  research  is  certainly  warranted  in  the  area  of 
decision-making  aids.  Some  recommendations  for  further 
study  are  as  follows: 

—  Conduct  similar  research  allowing  for  the  time  required 
to  learn  the  DSS. 
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—  Conduct  similar  research  with  a  more  complex  scenario. 

—  Conduct  field  (action)  research  with  a  real-world 
organization  dealing  with  nonprogrammed  decisions. 

—  Conduct  research  designed  to  examine  the  greater 
confidence  and  understanding  of  those  subjects  who  used  the 
DSS .  If  this  confidence  and  understanding  can  be  passed  on 
to  higher  levels  of  review,  then  the  use  of  the  DSS  would 
have  an  overall  positive  effect  on  the  source  selection 
process  as  well  as  on  further  complex  decisions. 

—  Conduct  research  into  the  AHP  and  its  possible  uses  as  a 
satisficing  or  bounded  rationality  strategy. 

—  Conduct  research  aimed  at  affecting  the  effort 
relationships  among  strategies  (rather  than  decision 
quality)  as  suggested  by  Todd  and  Adams. 

This  research  revealed  positive  effects  associated  with 
using  computer-aided  decision  support  systems.  It  also 
revealed  a  lack  of  effect  on  decision  quality  when  facing  a 
semi-structured  problem.  Clearly,  more  research  is  needed 
into  this  vital  area  of  emerging  technology. 
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Appendix  A:  Case  Study  and  Survey 


The  following  is  a  case  study  designed  to  test  your  ability 
to  make  decisions.  The  decision  at  hand  is  that  of  a 
government  source  selection.  You  will  be  presented  with  the 
situation,  the  government  technical  evaluation,  the 
government  cost  evaluation,  the  government  management 
evaluation  and  a  list  of  the  issues  that  the  government 
feels  are  important  to  the  decision.  You  will  then  be  asked 
to  make  a  decision  as  to  which  of  the  involved  contractors 
"best"  meet  the  criteria  that  the  government  has  deemed 
important  in  this  situation. 

Please  do  not  discuss  this  decision  with  anyone  else  in  your 
group.  Also,  do  not  discuss  the  method  by  which  you  intend 
to  make  your  decision  with  anyone  else. 

There  is  a  short  survey  that  you  are  asked  to  complete  at 
the  end  of  the  exercise.  Please  be  as  honest  as  you  can  in 
the  completion  of  the  survey.  ALTHOUGH  THERE  IS  A  "SCHOOL" 
ANSWER,  THIS  IS  NOT  A  "GRADED"  EXERCISE. 

You  must  complete  the  questionnaire  in  its  entirety  before 
you  leave  the  exercise.  You  will  not  be  allowed  to  leave 
until  that  time. 


GOOD  LUCK! 


NAME: 
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On  May  23  19  8X,  the  lab  at  Anywhere  AFB  released  a 
request  for  proposals  for  their  magnetic  fuse  program. 
Response  to  the  RFP  was  encouraging.  The  following 
contractors  submitted  proposals  (in  alphabetical  order) : 

a.  Deftoo  Ltd. 

b.  Electro  Tech 

c.  Fuses  'R  Us 

d.  Fusetech  Inc. 

e.  Magnetic  Technologies  Inc. 

f.  Smith  &  Jones  Co. 

g.  TECHCO 

An  initial  evaluation  was  performed  and  all  offerors  were 
considered  to  have  a  reasonable  chance  to  receive  the  award. 
You  are  provided  in  the  following: 

a.  a  list  of  the  evaluation  criteria  and  their  relative 

importance . 

b.  the  government  technical  evaluation 

c.  the  government  management  evaluation 

d.  the  government  cost  evaluation 

With  this  information  the  final  decision  must  be  made  as 
to  who  will  build  the  Magnetic  Fuse  for  Anywhere  AFB.  Keep 
in  mind  that  the  evaluation  criteria  that  you  have  are  those 
that  were  published  in  the  RFP  and  must  be  adhered  to.  A 
decision  not  based  on  these  criteria  will  most  likely  be 
protested  resulting  in  unacceptable  lost  time  and  money  for 
the  government. 

You  as  the  decision-maker  must  choose  amongst  the 
competing  contractors.  .  . 
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1.  EVALUATION  CRITERIA 

a.  General  Considerations 

Proposals  will  be  evaluated  on  a  subjective 
basis  for  their  conformance  with  the  terms  and  conditions  of 
the  solicitation. 

b.  General  Basis  for  Contract  Award 

(1)  The  Government  contemplates  awarding  one  (1) 
contract  to  the  offeror  whom  the  Government  determines  can 
accomplish  the  requirements  set  forth  in  this  RFP  in  a 
manner  most  advantageous  to  the  Government.  The  Government 
reserves  the  right  to  make  no  awards  at  all.  The  Government 
also  reserves  the  right  to  award  a  contract  at  other  than 
the  lowest  price  after  consideration  of  all  factors. 

(2)  This  is  a  technical  competition  with  cost  and 
management  considered  subordinate.  Offerors  are  encouraged, 
however,  to  perform  technical-cost  tradeoffs  to  achieve  a 
balance  where  the  proposed  cos^  must  be  entirely  compatible 
with  the  technical  and  managem.  t  proposals.  No  advantage 
will  accrue  to  an  offeror  who  submits  an  unrealistically  low 
cost  proposal.  Accordingly,  the  offeror's  proposal  may  be 
penalized  during  the  evaluation  to  the  degree  that  the 
estimated  cost  is  unrealistically  low. 

(3)  An  alternate  proposal  will  be  evaluated  in  the 
same  manner  as  a  proposal  that  meets  the  stated  basic 
proposal  requirements  or  the  stated  evaluation  criteria 
herein  and  in  accordance  with  the  Section  L  provision 
entitled,  "Alternate  Proposals." 

c.  Specific  Areas  of  Evaluation 

(1)  The  specific  areas  of  evaluation  shown  in 
descending  order  of  importance  are  as  follows: 

(a)  Technical  -  is  considered  one  and  one  half 
times  as  important  as  management  or  cost/price. 

(b)  Management  -  is  considered  equal  to 

cost/price. 

(c)  Cost/Price  -  is  considered  equal  to 

management. 

(2)  Technical  Area:  The  items  and  factors  to  be 
evaluated  in  this  area  are  listed  below.  The  items  listed 
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below  are  of  equal  importance.  The  order  of  importance  of 
the  factors  within  each  item  is  addressed  at  the  item  level. 


(a)  Soundness  of  Approach:  The  Offeror's 
proposal  shall  include  an  outline  of  the  main  problem  areas 
and  the  general  approach  used  to  solve  these  problems.  An 
evaluation  of  the  various  methods  considered  should  be 
presented  including  specific  experience  in  using  these 
methods  and  justifications  of  the  method  selected.  The 
evaluation  factors  are  listed  below  and  are  equally 
weighted. 

1 .  Approach  to  defining  target 

signatures. 

2 .  Approach  to  enhancement  of  signal-to- 

noise  ratio. 

3 .  Approach  to  signal  processing. 

(b)  Understanding  the  Problem:  The  proposal 
must  clearly  show  that  the  offeror  recognizes  all  the 
technical  requirements,  scope,  and  unique  problems 
associated  with  this  program.  Unclear,  inconsistent, ,  and 
incomplete  technical  information  will  be  interpreted  as  a 
lack  of  understanding  on  the  proposer's  part.  The  following 
evaluation  factors  are  weighted  equally. 

1 .  Plating  and  coating  of  fibers  and/or 
magnetostrictive  fibers. 

2 .  Mine  applications 


(c)  Compliance  with  Requirements:  A  response 
to  each  specific  requirement  in  the  statement  of  work  will 
be  evaluated.  Any  interpretations,  deviations,  and 
exceptions  shall  be  clearly  stated.  The  offeror  should 
describe  how  he  proposes  to  comply  wit  requirements.  The 
evaluation  factors  listed  below  are  equally  weighted: 

1 .  Schedule 

2 .  Laboratory  tests 

3 .  Field  test  support 
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(d)  Special  Technical  Factors:  The  offeror's  proposal 
should  be  responsive  to  the  factors  listed  below.  They  are 
equally  weighted. 

1 .  Growth  options 

2 .  Specific  experience 


(3)  Management  Area:  The  items  to  be  evaluated  in 
this  area  are  listed  in  descending  order  of  importance.  The 
order  of  importance  of  the  factors,  if  any,  within  each  item 
is  addressed  at  the  item  level. 

(a)  The  management  proposa.1  for  the  program 
must  be  complete  and  describe  procedures  for  identification, 
documentation,  and  control  during  design,  programming,  test 
and  validation  phases  of  the  program. 

(b)  The  program's  master  planning  schedule 
must  be  complete,  reasonable,  identify  each  task  and  clearly 
integrate  all  facets  of  the  proposed  program. 

(c)  The  organization  structure  should  show 
lines  of  authority/responsibility  and  communication  within 
the  company.  The  program  manager's  position  in  the 
organization  must  be  clearly  established  and  indicate  he  has 
the  authority  and  responsibility  to  successfully  accomplish 
the  project.  All  subcontractors  must  be  identified  and  the 
scope  of  their  responsibility  defined. 

(4)  Cost  Area:  The  offeror's  cost  proposal  will 
not  be  rated  or  scored,  but  will  be  reviewed  for 
reasonableness,  realism,  completeness,  and  continuity. 

Direct  material  and  direct  labor  hours  are  of  equal 
importance,  and  are  considered  twice  as  important  as  travel 
and  other  direct  costs. 

(a)  Direct  Material:  The  contractor's 
proposal  will  be  reviewed  to  determine  the  reasonableness  of 
the  bill  of  materials.  The  completeness  and  detail  of  the 
list  will  be  examined. 

(b)  Direct  Labor  Hours  and  Labor  Mix:  The 
total  hours  proposed  and  the  mix  of  skilled  labor  will  be 
examined  in  detail.  Salaries  will  be  verified  through 
audits. 
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(c)  Travel:  Travel,  if  any,  will  be  reviewed 
for  the  necessity  of  the  trips  and  the  costs  involved. 

(d)  Other  Direct  Costs:  It  is  conceivable 
that  the  contractor  will  require  ODCs.  The  nature  and 
explanation  of  these  costs  will  be  reviewed  for 
appropriateness . 
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Results  of  Technical  Proposal  Evaluation,  Magnetic  Fuse 


1.  The  technical  proposals  received  on  the  subject  RFP  have 
been  evaluated. 

2.  The  following  firms  submitted  proposals  that  are 
acceptable  on  a  technical  basis  and  are  recommended  for 
consideration . 

3.  Discussion  of  proposals  which  are  acceptable. 


a.  Deftoo  Ltd. 

(1)  Soundness  of  Approach.  Deftoo  Ltd. 's  approach 
is  based  on  requirements  and  preliminary  design  resulting 
from  previous  programs  as  well  as  a  consideration  of  trade¬ 
offs.  They  realize  the  Mach-Zehnder  configuration  as  being 
well  suited  for  high  sensitivity  field  sensing.  Refer  to 
Target  Activated  Munitions  Sensors  programs  and  reports  for 
high  signature  analysis  expertise.  Various  techniques  for 
maximizing  the  signal-to-noise  ratio  are  discussed  with  an 
active  interferometer  scheme  being  chosen.  All  approaches 
examined  are  considered  to  exceed  government  standards.  For 
this  purpose,  Deftoo  Ltd.  proposes  to  experiment  with 
various  signal  processing  and  analysis  techniques  during 
laboratory  and  field  tests. 

(2)  Understanding  the  Problem.  The  proposal  shows 
a  very  god  understanding  of  the  performance  and  design 
requirements  for  the  overall  MF.  Their  continuous  fiber 
plating  technology  is  optimized.  Deftoo  Ltd.  indicates 
current  work  in  various  fiber  optic  sensor  developments. 

The  proposal  shows  a  full  understanding  of  the  issues  for 
mine  applications. 

(3)  Compliance  with  Requirements.  The  contractor 
shows  an  acceptable  program  schedule.  Laboratory  and  field 
tests  are  not  specifically  defined,  but  parameters  to  be 
characterized  and  general  vehicle  types  are  not  noted.  This 
is  considered  a  weakness. 

(4)  Special  Technical  Factors.  While  their  fiber 
plating  technology  and  magnetic  knowledge  allows  for 
practical,  easy-to-fabricate,  magnetic  fuse  design,  growth 
potential  is  limited.  The  contractor  has  a  production 
plating  systems  which  provides  continuous  automated 
processes  for  making  the  sensor.  This  indicates  an  adequate 
experience  level. 
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b.  Electro  Tech 

(1)  Soundness  of  Approach.  Electro  Tech  proposes 
to  develop  a  magnetometer  based  on  the  standard  Mach-Zehnder 
technique  outlined  in  the  many  trade  journals.  This  fiber 
optic  interferometer  technique  is  based  on  the  demonstrated 
principle  that  if  one  branch  of  a  two  branch  fiber  bundle  is 
coated  with  magnetostriction  material  or  glued  to  a  bulk 
material  such  as  metglas,  then  a  magnetic  disturbance  will 
result  in  an  optical  fringe  pattern  which  is  readily 
detected.  Electro  Tech  proposes  to  glue  the  fiber  to 
metglas.  This  technique  does  not  appear  to  offer  advantages 
of  the  other  techniques  due  to  stability  problems  and  the 
potentiality  of  high  background  noise;  however,  since  the 
optical  wavelength  is  very  small  (.8  -  .9  micron)  and  the 
fibers  can  be  made  very  long  in  terms  of  wavelength,  the 
device  can  be  extremely  sensitive.  Also,  the  "cook  book" 
approach  prepared  by  Electro  Tech  does  not  reflect  any 
innovative  ideas  as  to  improvements  in  the  selections  of 
fiber/coatings,  signal  processing  techniques,  or  signal-to- 
noise  ratio  enhancements. 

(2)  Understanding  -he  Problem.  Electro  Tech 
demonstrated  an  understanding  of  the  problem  by  presenting  a 
straight  forward  approach.  Questionably,  Electro  Tech 
avoided  the  issue  of  fiber  plating  by  proposing  to  glue 
sensor  fibers  directly  to  metglas.  The  sensitivity  of  this 
technique  may  be  less  than  plating  fibers  with  a 
magnetostriction  material  such  as  nickel.  Electro  Tech's 
proposal  did  not  indicate  the  in-house  fabrication  of  a 
breadboard  which  would  have  provided  data  as  to  the 
sensitivity  of  Electro  Tech's  Metglas  concept.  It  appears 
that  Electro  Tech's  approach  explored  the  use  of  acceptable 
optical  sensor  technology.  No  reference  to  mine 
applications  was  made. 

(3)  Compliance  With  Reguirements.  The  proposal 
indicated  compliance  with  requirements  except  for  the 
requirement  that  the  sensor  have  a  frequency  response  from 
DC  to  5000  Hz.  The  contractor  states  that  the  lowest 
frequency  of  sensor  operation  is  fixed  by  the  relative 
velocity  of  the  target  and  does  not  necessarily  extend  down 
to  DC  in  an  absolute  sense.  The  contractor  seemed  to  be 
hedging  on  meeting  this  requirement.  This  indicates  a  weak 
preliminary  laboratory  test  schedule.  No  field  tests  were 
indicated. 
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(4)  Special  Technical  Factors.  The  contractor  is 
weakest  in  the  growth  options  and  specific  experience 
factors.  Although  the  SOW  does  not  require  that  the  sensor 
determine  range  and  angle  to  target.  Electro  Tech  appeared 
to  unnecessarily  exclude  considerations  of  the  sensor's  long 
term  ability  to  measure  target  range  and  bearing.  The 
sensor  may,  therefore,  not  have  as  high  a  potential  for 
growth.  The  proposal  did  not  reflect  any  specific 
experience  in  the  fabrication  and  testing  of  magnetic  fuses. 

c.  Fuses  *R  Us 

(1)  Soundness  of  Approach.  Fuses  'R  Us  proposes  to 
use  their  Mach-Zehnder  interferometer.  One  leg  of  the 
interferometer  is  a  reference  path  and  the  other  is  immersed 
in  a  ferrofluid  bath.  By  choosing  a  ferrofluidic  bath  as 
the  sensing  medium,  the  long  term  stability,  and  therefore, 
successful  implementation  of  the  sensor  in  the  field  is 
questionable.  With  this  approach  signal  processing  and 
target  signature  definition  is  hampered. 

(2)  Understanding  the  Problem.  Fuses  'R  Us 
demonstrated  an  understanding  of  the  problem  by  recognizing 
several  potential  techniques  for  a  demodulation  scheme. 

They  proposed  a  detailed  investigation  of  the  approaches  as 
part  of  the  program.  For  mine  applications,  Fuses'R  Us  will 
use  a  battery  that  is  adequate  for  a  30-day  remote  operation 
in  the  pulsed  mode.  The  action  of  coating  the  reference 
fiber  with  nickel  is  questionable.  It  may  invalidate  the 
sensor  since  both  the  "reference"  and  sensing  fibers  would 
then  react  to  magnetic  field  fluctuation. 

(3)  Compliance  with  Requirements.  The  contractor 
proposes  to  meet  all  requirements  set  forth  in  the  RFP 
without  exception.  Fuses  'R  Us  expects  to  complete  the 
program  in  20  months.  Special  test  equipment  may  need  to  be 
purchased  to  conduct  such  tests  as  immunity  of  RFI  and  EMI 
interference.  These  are  necessary  to  complete  both  lab  and 
field  tests. 

(4)  Special  Technical  Factors.  The  contractor  did 
not  evidence  any  specific  experience  with  magnetic  sensors 
other  than  an  interferometer.  Also,  this  type  of  sensing 
scheme  appears  to  limit  the  growth  options  of  the  sensor.  * 
Fuses'R  Us  proposal  contains  an  innovative  approach. 

However,  because  of  the  unknowns  stated  above  which  appear 
inherent  in  this  approach,  we  feel  that  unless  they  can  be 
satisfactorily  resolved  prior  to  award,  the  risk  to  pursue 
this  approach  would  be  unacceptably  high. 
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d.  Fusetech  Inc. 

(1)  Soundness  of  Approach.  Fusetech  Inc.  has 
proposed  an  outstanding  approach  to  solving  the  problems  of 
the  magnetic  fuse.  The  resultant  design  provides  for 
operation  at  the  point  of  maximum  sensitivity,  permits  the 
use  of  a  permanent  magnet  bias,  and  is  extremely  rugged.  It 
proposes  maximizing  the  signal-to-noise  ratio  by  use  of  a 
very  unique  low  noise  interferometer  developed  at  Fusetech 
Inc.  by  using  divided  Fabry-Perot  fiber  optic  cells,  common¬ 
mode  rejection,  matched  path  lengths  and  a  multimode  laser 
source.  To  define  target  signatures  Fusetech  Inc.  proposes 
to  perform  a  comprehensive  search  of  classified  target 
characteristic  literature.  This  is  considered  an  excellent 
approach.  This  approach  will  lead  to  highly  efficient 
signal  processing. 

(2)  Understanding  the  Problem.  The  contractor 
shows  an  excellent  understanding  of  the  desired  performance 
and  requirements  of  Magnetic  Fuse  (MF) .  For  example,  they 
recognize  that  phase  modulation  represents  the  most 
sensitive  approach  to  plating  and  coating.  In  mine 
applications,  they  propose  to  use  the  MF  to  isolate  nearby 
targets  and  determine  the  point  of  nearest  approach  in  order 
to  detonate  the  mine  at  the  optimum  point. 

(3)  Compliance  with  Requirements.  The  contractor 
proposes  to  comply  with  all  requirements  of  the  RFP.  The 
proposal  includes  an  excellent  breakdown  of  what  will  be 
done  on  the  various  levels  of  testing  to  verify  compliance 
with  requirements.  Results  of  the  preliminary  experiments, 
to  insure  proper  selection  of  the  fiber  coating  alloy 
material,  will  be  thoroughly  reviewed  with  sponsor  before 
selection  is  made.  Additional  experiments  will  then  be 
conducted  as  they  become  appropriate.  Fusetech  Inc.  has 
their  own  staff  of  experienced  laboratory  and  field  test 
personnel.  This  experienced  laboratory  and  field  test 
capability  will  significantly  reduce  program  risk  by  early 
discovery  of  potential  design  deficiencies. 

(4)  Special  Technical  Factors.  The  contractor 
states  that  the  sensor  has  considerable  growth  potential  in 
that  with  only  sensor  modification,  the  sensor  cold  be  used 
in  a  multi-influence  sensor.  Active  homodyning  detection 
will  be  employed  in  order  to  provide  the  maximum  dynamic 
range.  Exclusive  of  the  power  source,  the  proposed  sensor 
is  compatible  with  the  volume  requirement.  The  contractor 
has  exceptional  experience  in  optical  research  with  Navy  and 
Health  Institute  applications. 
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e.  Magnetic  Technologies  Inc. 

(1)  Soundness  of  Approach.  Magnetic  Technologies 
Inc.  proposes  a  fiber  optic  magnetic  sensor  utilizing  a 
stabilized  laser  diode  source,  a  Mach-Zehnder 
interferometer,  and  single  mode,  polarity  preserving  optical 
fibers.  Their  approach  is  straight  forward  and  adequately 
details  the  necessary  tasks  needed  for  defining  target 
signatures  and  signal  processing  techniques  to  enhance  the 
signal-to-noise  ratio. 

( 2 )  Understanding  the  Problem.  The  contractor ' s 
approach  demonstrated  a  thorough  understanding  of  the  tasks 
unique  to  this  program.  Magnetic  Technologies  Inc.  is 
competent  in  the  area  of  coating  systems  and  procedures  for 
sputter  and  vapor  phase  deposition.  Sputtering  generally 
promotes  excellent  interfacial  cohesion  and  is  often 

px.  ierred  when  feasible.  Magnetic  Technologies  Inc. 
proposes  to  run  sputter  depositions  of  both  pure  nickel  and 
iron-nickel-cobalt  alloys,  which  may  prove  to  offer 
advantages  over  use  of  pure  nickel.  Such  coatings  are 
generally  superior  to  electroplated  films  and  should  provide 
much  better  magnetostrictive  behavior.  Magnetic 
Technologies  Inc.  prefers  sputtering  to  electroplating.  The 
prospect  of  using  metglas  ribbon  was  ruled  out  because  it 
might  interfere  with  transducer  design  and  fabrication. 
Magnetic  Technologies  Inc.  has  prior  knowledge  and 
experience  with  mine  applications  through  endeavors  in  the 
Wide  Area  Side  Penetrating  (WASPM)  off-route  mine  for  target 
classification  from  vehicular  acoustic  emissions.  Magnetic 
Technologies  Inc. 1  s  proposal  reflects  that  the  company  has 
made  substantial  efforts  to  expand  their  technical 
capabilities  to  enable  them  to  completely  develop  magnetic 
field  sensing  mines  as  well.  There  are  signal  processing 
similarities  involved  in  both  tasks  in  order  that  target 
signatures  be  defined  and  identified.  Furthermore ,  the 
design  and  fabrication  of  military  qualified  hardware 
entails  planning,  standards,  and  field  testing  which  will 
not  be  new  to  Magnetic  Technologies  Inc.  Magnetic 
Technologies  Inc.  has  had  a  significant  amount  of  experience 
in  the  area  of  fiber  optic  devices  and  mine  applications. 

(3)  Compliance  with  Requirements.  The  contractor 
proposes  to  accomplish  all  requirements  of  the  RFP.  The 
general  design  is  an  magnetic  sensor  with  c  stabilized  laser 
diode  source,  Mach-Zehnder  interferometer  and 
detector/signal  processor.  Magnetic  Technologies  Inc.  have 
their  own  highly  skilled  laboratories,  field  tests  equipment 
and  personnel. 
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(4)  Special  Technical  Factors.  The  contractor  has 
proposed  an  innovative  concept  for  coating  and  sputtering 
the  fibers  in  their  final  configuration  which  greatly 
enhances  the  practicality  and  affordability  of  the  sensors 
without  degrading  the  growth  potential .  The  contractor 
lacks  specific  experience  with  fiber  optic  magnetic  sensors, 
but  has  the  experience  with  other  sensors  and  signal 
processing  techniques  necessary  for  a  successful  program. 


f.  Smith  &  Jones  Co. 

(1)  Soundness  of  Approach.  This  proposal  is  based 
on  mechanical  interfacing  of  magnetostrictive  metal  alloys 
to  single  mode  optical  fibers.  Optical  mixing  is 
accomplished  by  use  of  a  Mach-Zehnder  interferometer  in  a 
simple  arrangement  with  a  polar imeter.  The  proposal 
considers  all  appropriate  types  of  fiber  and  coating 
materials,  including  high  birefringent  fibers  and  metglas 
nickel  or  deposited  nickel  subjected  to  magnetic  field 
annealing  cycle.  Target  signatures  will  be  defined 
primarily  by  doing  field  tests  at  Anywhere  AFB,  but  the 
contractor  emphasizes  the  use  of  field  tests  to  determine  or 
demonstrate  the  effective  sensing  range  against  the  moving 
targets  and  not  the  ability  to  discriminate  between  them. 
Signal-to-noise  ratio  is  maximized  through  consideration  of 
optimization  of  total  fiber  length  depending  upon  shot  noise 
introduced  and  its  effects  upon  sensitivity  for  various 
lengths  of  fiber.  It  has  been  determined  that  the  0.1  gamma 
requirement  can  be  achieved  given  the  maximum  laser  noise 
which  is  known  and  their  capabilities  to  adjust  circuit 
design  parameters.  Approach  to  signal  processing  is  still 
questionable  because  the  proposal  doesn't  specifically 
address  the  means  by  which  identification  and  definition  of 
various  target  signatures  will  be  accomplished  other  than 
through  interaction  with  Anywhere  AFB  following  preliminary 
field  tests. 

(2)  Understanding  the  Problem.  The  proposal 
identifies  no  exception  to  the  SOW  requirements.  The 
contractor  fully  understands  the  need  to  optimize  process 
parameters  to  fabricate  sensor  fiber  sensor  fiber  which 
exhibits  the  desired  magnetostrictive  behavior.  Plating  is 
not  addressed;  yet  vapor  deposition,  sputtering,  and 
adhesion  to  metglas  are,  and  these  are  of  primary  importance 
since  they  facilitate  better  control  magnetostrictive 
material  thickness.  The  contractor  is  quite  knowledgeable 
of  mine  technology  capable  of  integrating  sensor  technology 
in  general,  capable  of  integrating  the  various  components 
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including  the  piezoelectric  compensator,  beam 
splitter/polarizer,  dither  coil  and  diodes,  as  well  as  in 
the  design  and  fabrication  of  the  magnetostrictive 
transducer. 

(3)  Compliance  with  Requirements.  The  contractor 
shows  an  ambitious  compliance  with  the  schedule  proposing 
that  the  first  three  tasks  be  accelerated  relative  to  RFP 
schedule.  This  is  considered  optimal.  Judging  from  the 
technical  content  of  the  proposal,  one  may  conclude  that 
there  is  a  good  likelihood  of  the  contractor  meeting  the 
schedule  provided  Anywhere  AFB  personnel  manage  to  push  for 
timely  field  testing  so  that  a  few  design  iterations  may  be 
facilitated.  It  is  apparent  that  the  preliminary  designs 
will  serve  a  purpose  for  the  first  trials;  their  Mach- 
Zehnder  interferometer  meets  the  standard  for  comparison. 
Laboratory  testing  capabilities  appear  to  be  quite  adequate 
and  they  have  the  necessary  technical  expertise  to  structure 
an  adequate  testing  program.  Breadboard  design  and 
fabrication  are  adequate  except  that  redesign  and 

reconf iguration  may  have  to  be  performed  in  order  to  satisfy 
the  25  cm3  maximum  volume  requirement.  Batteries  are  not 
self-contained  according  to  the  breakdown  of  all  the 
elements  included.  Field  test  support  was  only  weakly 
defined  in  the  proposal. 

(4)  Special  Technical  Factors.  This  contractor 
exhibits  prior  and  on  going  experience  with  both  the  Navy 
and  the  Army  to  develop  a  ruggedized  fiber  optic 
magnetometer.  The  growth  potential  is  limited  to  the 
possible  incorporation  of  a  miniaturized,  multi-sensor  array 
into  munitions  for  identification  of  specific  targets. 
Practicality  of  the  proposed  magnetometer  appears  to  be 
adequate  based  upon  the  technical  considerations  addressed. 


g.  TECHCO 

(1)  Soundness  of  Approach.  The  contractor  proposed 
the  Mach-Zehnder  interferometer  approach  for  MF  design.  The 
contractor  has  done  an  exceptionally  thorough  job  in 
comparing  fiber  coatings  and,  finally,  choosing  amorphous 
metals  as  the  materials  of  choice  for  design  of  MF.  To 
define  target  signatures,  environmental  magnetic  noise  will 
be  studied.  Signal  processing  will  be  conducted  by  device 
signature  stu^.y  to  determine  the  best  method  for  detection 
and  thresho'  .  The  data  will  be  correlated  using  multi¬ 
sensor  inputs.  All  of  these  approaches  are  soundly  based  in 
Magnetic  Fuse  Theory. 
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(2)  Understanding  the  Problem.  The  contractor 
assumes  a  simple  total  field  magnetomer  is  the  current 
desirable  configuration,  while  considering  such  options  as 
multiaxis  sensor  and  magnetic  gradient  devices.  Mine 
applications  of  this  approach  are  extremely  limited. 

(3)  Compliance  with  Requirements.  The  proposal 
presents  general  considerations  and  a  brief  description  of 
each  test  to  be  carried  out.  Scheduling  of  the  tests  and 
program  are  acceptable. 

(4)  Special  Technical  Factors.  The  approach  to 
place  priorities  on  achievement  of  reliable  magnetic  field 
measurements  and  data  acquisition,  rather  than  on  real  time 
signal  processing  functions,  improves  the  evaluation  of 
magnetic  sensor  feasibility  and  allows  for  feature  design 
evaluation.  Growth  potential  is  high.  Specific  experience 
needed  to  carry  out  this  program  is  adequate.  No  further 
test  facilities  are  necessary. 
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Management  Evaluation 

a.  Deftoo  Ltd. 

Deftoo  Ltd.  did  not  submit  a  management  proposal. 
They  addressed  management  in  their  technical  discussion,  yet 
only  as  an  after  thought.  All  aspects  of  their  "management" 
must  be  considered  suspect. 

b.  Electro  Tech 

Electro  Tech  did  not  address  management.  This  must 
be  considered  unacceptable. 

c.  Fuses  'R  Us 

The  management  proposal  for  Fuses  'R  Us  must  be 
considered  outstanding.  It  is  clear  that  much  thought  was 
given  to  the  organization  of  the  program.  The  master  plan 
must  be  given  the  highest  marks. 

d.  Fusetech  Inc. 

Fusetech  Inc.  presents  only  a  rudimentary  management 
proposal.  Documentation  and  control  of  the  program  are  not 
addressed.  The  master  planning  schedule  is  good,  but 
incomplete.  The  organizational  structure  is  well  defined. 

e.  Magnetic  Technologies  Inc. 

Magnetic  Technologies  Inc.  proposed  a  good,  sound 
procedure  to  track  and  document  the  program.  Lines  of 
authority  are  clearly  defined.  The  master  planning  schedule 
was  adequate  but  could  be  improved. 

f.  Smith  &  Jones  Co. 

An  excellent  approach  to  documentation  and  control 
was  proposed.  The  master  planning  schedule  was  outstanding, 
however  with  two  lead  scientists  the  organizational 
structure  seems  questionable. 

g.  TECHCO 

The  management  scheme  of  TECHCO  is  considered  to  be 
optimal.  All  areas  of  control  and  documentation  are 
addressed.  The  organizational  structure  is  clearly  defined. 
An  excellent  "road  map"  of  the  program  is  prescribed  in  the 
master  planning  schedule. 
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Cost  Proposal  Evaluation 


1.  The  seven  cost  proposals  in  response  to  the  subject  RFP 
have  been  evaluated.  The  evaluations  are  given  below  with 
discussions. 

a.  Deftoo  Ltd. 

(1)  Direct  Material:  The  direct  materials  estimate 
is  very  high.  Questionable  costs  are  for  fibers  and  for 
miscellaneous  electronics  hardware  and  signal  processing 
electronics. 

(2)  Direct  Labor  Hours  and  Labor  Mix:  The  total 
hours  estimate  of  2374  appears  to  be  low.  The  hourly  rates 
are  reasonable.  Only  four  labor  categories  work  on  this 
project  with  a  student  aide  doing  740  hours  of  the  work  and 
a  level  II  profe  jsional  doing  814  hours.  This  is 
approximately  60%  of  the  wo_k  effort  which  is  not 
acceptable. 

(3)  Travel ;  Deftoo  Ltd.  proposes  money  for  travel 
but  identifies  no  details.  A  minimum  of  two  trips  to 
Anywhere  AFB  would  be  required.  The  dollar  amount  proposed 
for  these  trips  is  reasonable. 

(4)  Other  Direct  Costs;  Other  costs  include  sales 
and  technical  and  defense  systems.  The  contractor  did  not 
identify  what  these  are.  This  amount  appears  to  be 
excessive. 

b.  Electro  Tech 

(1)  Direct  Material:  The  list  of  required 
materials  was  not  provided.  They  also  show  no  cost  for 
material.  This  is  unacceptable. 

(2)  Direct  Labor  Hours  and  Labor  Mix:  The  total 
number  of  man-hours  proposed,  4680,  as  well  as  the  hour 
breakout  by  task  and  labor  category  are  considered 
reasonable  and  acceptable  to  accomplish  the  technical  tasks 
proposed. 

(3)  Travel ;  The  contractor  identified  four  travel 
requirements  but  gave  no  details.  The  most  costly  trip  is 
one  listed  under  task  six  for  field  test.  Though  the 
proposed  trip  is  necessary  the  cost  seems  unreasonable. 
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(4)  Other  Direct  Costs:  Other  direct  costs  were 
not  addressed  in  this  proposal.  This  is  unacceptable. 

c.  Fuses  'R  Us 

(1)  Direct  Material:  The  list  of  required 
materials  is  presented  in  detail  but  is  considered  high. 

The  quantity  listed  for  some  items  is  high.  The  quantities 
are  considered  unacceptable. 

(2)  Direct  Labor  Hours  and  Labor  Mix:  The  proposed 
4617  hours  of  direct  labor  is  considered  reasonable.  The 
senior  project  engineer  man-hours  represent  approximately 
one-half  of  the  total  hours  which  appear  to  be  a  reasonable 
mix. 


(3)  Travel ;  None.  This  is  no:  acceptable. 

(4)  Other  Direct  Costs;  Cost  of  a  Burroughs 
computer  rental  at  $12 5/hour  was  noted  in  the  general 
discussion  but  not  listed  on  the  cost  breakout.  No  other 
costs  were  addressed.  This  was  considered  reasonable. 

d .  Fusetech  Inc . 

(1)  Direct  Material:  The  direct  material  estimate 
is  high.  The  list  of  required  materials  is  presented  in 
detail  and  is  considered  acceptable. 

(2)  Direct  Labor  Hours  and  Labor  Mix;  The  total 
hours  estimate  of  1320  man-days  (10,560  hours)  proposed  is 
high.  The  man-day  breakout  by  task  and  labor  category  is 
unreasonable  and  not  acceptable  to  accomplish  the  technical 
tasks.  It  contains  too  many  hours  and  too  many  people 
allotted  to  trivial  tasks.  Salary  rates  and  time  for  labor 
categories  could  be  cut  from  1/3  to  1/2  with  some  of  the 
categories,  data  taker,  jr  physicist,  and  order  processor, 
eliminated  completely. 

(3)  Travel ;  Five  trips  to  Anywhere  AFB  for  sponsor 
reviews  are  proposed.  Two  are  considered  adequate. 

(4)  Other  Direct  Costs:  The  contractor  proposes 
other  direct  costs.  As  there  is  no  back-up  for  these  costs, 
they  roust  be  considered  high. 

e.  Magnetic  Technologies  Inc. 

(1)  Direct  Material:  The  direct  material  costs 
seems  reasonable.  The  major  cost  items  are  for  material 
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(tooling  and  fixtures,  20) ,  six  optical  splitters  and  a 
transducer. 


(2)  Direct  Labor  Hours  and  Labor  Mix:  The  total 
hours  estimate  of  5094  is  very  reasonable  with  engineers 
doing  2752  hours  of  the  total  work.  The  other  hours  are 
divided  between  16  other  categories  of  work.  Descriptions 
of  each  category  are  provided.  The  burden  rate  is  very 
reasonable. 

(3)  Travel ;  The  proposed  travel  includes  two  trips 
to  Anywhere  AFB.  This  is  considered  optimal. 

(4)  Other  Direct  Costs:  Additional  costs  were 
proposed  for  reproduction  and  photographs.  Excellent 
rationale  is  provided. 


f.  Smith  &  Jones  Co. 

(1)  Direct  Material:  The  estimate  is  high.  The 
cost  for  electrical  and  mechanical  fabrication  and  packaging 
alone  is  unreasonable. 

(2)  Direct  Labor  Hours  and  Labor  Mix:  The  total 
hours  of  5936  is  somewhat  high  for  the  7  labor  categories 
provided.  Labor  categories  such  as  senior  scientist  I,  II, 
and  III  seem  excessive. 

(3)  Travel :  Smith  &  Jones  Co.  proposes  nine 
trips  -  five  to  Washington  D.C.  and  four  to  Anywhere  AFB, 
all  for  technical  discussions.  No  explanation  was  given  for 
the  trips  to  Washington  D.C.  These  are  unnecessary. 

(4)  Other  Direct  Costs:  Other  costs  are  high  and 
include  money  for  publication.  The  other  money  is  included 
in  the  task  breakout,  but  is  not  defined.  This  appears  to 
be  high. 

g .  TECHCO 

(1)  Direct  Material;  The  direct  materials  cost  was 
high  with  their  choice  of  two  laser  diodes  costing  in  excess 
of  estimates.  There  is  a  list  of  required  materials,  and  it 
is  considered  high. 

(2)  Direct  Labor  Hours  and  Labor  Mix:  The  total 
hours  estimate  of  4,432  is  reasonable.  The  total  direct 
labor  cost  is  reasonable.  The  labor  mix  is  high  on 
unskilled  labor. 
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(3)  Travel ;  Travel  costs  include  seven  trips  to 
Anywhere  AFB.  Reasons  supporting  the  trips  were  not 
included.  Seven  trips  to  Anywhere  AFB  seem  unreasonable. 

(4)  Other  Direct  Costs:  Other  direct  costs  are  for 
reprographics.  This  is  reasonable  and  includes  all  reports 
and  drawings. 
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Please  answer  the  following  questions: 

1.  What  is  the  highest  level  of  education  that  you  have 
completed? 

1  HIGH  SCHOOL  3  SOME  COLLEGE  (1-3  YRS) 

2  VOCATIONAL/TECHNICAL  SCHOOL  4  COLLEGE  GRADUATE 

5  SOME  POST-GRADUATE  OR  PROFESSIONAL 

6  POST-GRADUATE  OR  PROFESSIONAL  DEGREE 

2.  What  was  your  major  field/area  of  study  for  your  highest 
level  of  education?  (ANSWER  EVEN  IF  YOU  DID  NOT  RECEIVE  THE 
DEGREE  OR  COMPLETE  THE  PROGRAM) 

1  BUSINESS  4  ENGINEERING 

2  LAW  5  PUBLIC  ADMINISTRATION 

3  LIBERAL  ARTS  6  OTHER  _ 

3 .  How  many  formal  acquisition-related  training  courses  have 
you  taken  since  the  beginning  of  your  career?  (Circle  the 
number  of  your  answer) 

1  NONE  34-6  5  11  OR  MORE 

21-3  47  -  10 

4.  How  many  courses  have  you  taken  in  your  expertise  or 
related  disciplines? 


5.  How  many  years  of  experience 
do  you  have? 

1  LESS  THAN  ONE  YEAR 

2  1  YEAR 

3  2  YEARS 


in  your  present  career  field 

4  3  YEARS 

5  4  YEARS 

6  5  YEARS  AND  OVER 
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6.  Please  indicate  whether  you  have  been  designated  as  one 
or  more  of  the  following: 

1  CERTIFIED  PROFESSIONAL  CONTRACTS  MANAGER 

2  CERTIFIED  ASSOCIATE  CONTRACTS  MANAGER 

3  CERTIFIED  PURCHASING  MANAGER 

4  OTHER  _ 

5  NOT  CERTIFIED. 

7.  Time  Finished  _ . 

8.  Which  Contractor  did  you  choose? 


9.  Why  did  you  choose  this  contractor?  (Please  use  the  back 
of  this  page  if  you  require  additional  space.) 


10.  If  you  had  had  more  time,  could  you  have  made  a  "better" 
decision? 
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11.  Would  you  classify  this  decision  as  structured 
(programmed)  or  unstructured  (non-programmed) ? 


How  strongly  do  you  agree/disagree  with  the  following 
statements? 


12 .  I  understood  the  process  by  which  the  correct  contractor 
was  to  be  chosen. 

1  STRONGLY  2  MILDLY  3  NEUTRAL  4  MILDLY  5  STRONGLY 

AGREE  AGREE  DISAGREE  DISAGREE 


13.  I  found  the  decision-making  process  easy. 

1  STRONGLY  2  MILDLY  3  NEUTRAL  4  MILDLY  5  STRONGLY 
AGREE  AGREE  DISAGREE  DISAGREE 


14.  I  am  confident  that  I  made  the  right  decision. 

1  STRONGLY  2  MILDLY  3  NEUTRAL  4  MILDLY  5  STRONGLY 
AGREE  AGREE  DISAGREE  DISAGREE 


15.  How  did  you  feel  about  this  process? 


1  FRUSTRATED 

2  ANGRY 

3  JOY 

4  RELAXED 


5  APATHY 

6  ANXIETY 

7  HARRIED 

8  OTHER 
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